Re: [VOTE] KIP-55: Secure quotas for authenticated users

[Gwen Shapira]

+1

Thanks for working through this, Rajini.

On Thu, Jul 7, 2016 at 7:12 AM, Rajini Sivaram
 wrote:
> I would like to initiate voting for KIP-55 (
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-55%3A+Secure+Quotas+for+Authenticated+Users).
> Since the KIP has changed quite a lot since the last vote, we will discard
> the previous vote and start this new voting thread.
>
> KIP-55 extends the existing client-id quota implementation to enable secure
> quotas for multi-user environments. The KIP proposes a flexible, unified
> design that supports quotas at ,  or 
> levels. It retains compatibility with the existing  quotas when
> new user level quotas are not configured.
>
> Thank you...
>
>
> Regards,
>
> Rajini

Re: [DISCUSS] KIP-4 ACL Admin Schema

[Gwen Shapira]

Thank you, Grant. This is lovely :)

Few comments / requests for clarifications below:


>> ListAcls Request (Version: 0) => principal resource
>>   principal => NULLABLE_STRING
>>   resource => resource_type resource_name
>> resource_type => INT8
>> resource_name => STRING

I am a bit confused about specifying resources.
resource_type is something like "TOPIC" and resource_name is a name of
a specific topic?
Can you clarify a bit more about the use here? Can I have regexp? Can
I leave resource_name empty and have the ACLs for everything in a
resource type?
Also, can you describe the interaction between principal and resource?
I assume that if I specify both, I get all ACLs for a principal for
the resources specified, but just making sure :)


>> Alter ACLs Request
>>
>>3. ACLs with a delete action will be processed first and the add
>>action second.
>>1. This is to prevent confusion about sort order and final state when
>>   a batch message is sent.
>>   2. If an add request was processed first, it could be deleted right
>>   after.
>>   3. Grouping ACLs by their action allows batching requests to the
>>   authorizer via the Authorizer.addAcls and Authorizer.removeAcls calls.

I like this decision

>>  - I suggest this be addressed in KIP-50 as well, though it has
>>  some compatibility concerns.

Isn't KIP-50 itself one gigantic compatibility concern? I don't see
how your suggestions make it any worse...

Re: [DISCUSS] KIP-66 Kafka Connect Transformers for messages

[Gwen Shapira]

I used to work on Apache Flume, where we used to allow users to filter
messages completely in the transformation and then we got rid of it,
because we spent too much time trying to help users who had "message
loss", where the loss was actually a bug in the filter...

What we couldn't do in Flume, but perhaps can do in the simple
transform for Connect is the ability to route messages to different
topics, with "null" as one of the possible targets. This will allow
you to implement a dead-letter-queue functionality and redirect
messages that don't pass filter to an "errors" topic without getting
rid of them completely, while also allowing braver users to get rid of
messages by directing them to "null".

Does that make sense?

Gwen

On Thu, Jul 14, 2016 at 8:33 PM, Nisarg Shah  wrote:
> Thank you for your inputs Gwen and Michael.
>
> The original reason why I suggested a set based processing is because of the 
> flexibility is provides. The JIRA had a comment by a user requesting a 
> feature that could be achieved with this.
>
> After reading Gwen and Michael's points, (I went through the documentation 
> and the code in detail) and agree with what you have to say. Also, fewer 
> guarantees make what I had in mind less certain and thus simplifying it to a 
> single message based transformation would ensure that users who do require 
> more flexibility with the transformations will automatically “turn to" Kafka 
> Streams. The transformation logic on a message by message basis makes more 
> sense.
>
> One usecase that Kafka Connect could consider is adding or removing a message 
> completely. (This was trivially possible with the collection passing). Should 
> users be pointed towards Kafka Streams even for this use case? I think this 
> is a very useful feature for Connect too, and I’ll try to rethink on the API 
> too.
>
> Removing a message is as easy as returning a null and having the next 
> transformer skip it, but adding messages would involve say a queue between 
> transformers and a method which says “pass message” to the next, which can be 
> called multiple times from one “transform” function; a variation on the chain 
> of responsibility design pattern.
>
>> On Jul 12, 2016, at 12:54 AM, Michael Noll  wrote:
>>
>> As Gwen said, my initial thought is that message transformations that are
>> "more than trivial" should rather be done by Kafka Streams, rather than by
>> Kafka Connect (for the reasons that Gwen mentioned).
>>
>> Transforming one message at a time would be a good fit for Kafka Connect.
>> An important use case is to remove sensitive data (such as PII) from an
>> incoming data stream before it hits Kafka's persistent storage -- this use
>> case can't be implemented well with Kafka Streams because, by design, Kafka
>> Streams is meant to read its input data from Kafka (i.e. at the point when
>> Kafka Streams could be used to removed sensitive data fields the data is
>> already stored persistently in Kafka, and this might be a no-go depending
>> on the use case).
>>
>> I'm of course interested to hear what other people think.
>>
>>
>> On Tue, Jul 12, 2016 at 6:06 AM, Gwen Shapira  wrote:
>>
>>> I think we need to restrict the functionality to one-message-at-a-time.
>>>
>>> Basically, connect gives very little guarantees about the size of the set
>>> of the composition (you may get same messages over and over, mix of old and
>>> new, etc)
>>>
>>> In order to do useful things over a collection, you need better defined
>>> semantics of what's included. Kafka Streams is putting tons of effort into
>>> having good windowing semantics, and I think apps that require modification
>>> of collections are a better fit there.
>>>
>>> I'm willing to change my mind though (have been known to happen) - what are
>>> the comments about usage that point toward the collections approach?
>>>
>>> Gwen
>>>
>>> On Mon, Jul 11, 2016 at 3:32 PM, Nisarg Shah  wrote:
>>>
 Thanks Jay, added that to the KIP.

 Besides reviewing the KIP as a whole, I wanted to know about what
>>> everyone
 thinks about what data should be dealt at the Transformer level.
>>> Transform
 the whole Collection of Records (giving the flexibility of modifying
 messages across the set) OR
 Transform messages one at a time, iteratively. This will restrict
 modifications across messages.

 I’ll get a working sample ready soon, to have a look. There were some
 comments about Transformer usage that pointed to the first approach,
>>> which
 I prefer too given the flexibility.

> On Jul 11, 2016, at 2:49 PM, Jay Kreps  wrote:
>
> One minor thing, the Transformer interface probably needs a close()
 method
> (i.e. the opposite of initialize). This would be used for any
>>> transformer
> that uses a resource like a file/socket/db connection/etc that needs to
 be
> closed. 

[jira] [Commented] (KAFKA-3960) Committed offset not set after first assign

[Alexey Romanchuk (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15378874#comment-15378874
 ] 

Alexey Romanchuk commented on KAFKA-3960:
-

Anyone?

> Committed offset not set after first assign
> ---
>
> Key: KAFKA-3960
> URL: https://issues.apache.org/jira/browse/KAFKA-3960
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.10.0.0
>Reporter: Alexey Romanchuk
>Priority: Blocker
>
> Committed offset did not set after first assign. Here it is minimal example 
> (scala):
> {code}
>   val props = new Properties()
>   props.put("bootstrap.servers", "localhost:9092")
>   props.put("client.id", "client1")
>   props.put("group.id", "client1")
>   props.put("enable.auto.commit", "false")
>   props.put("key.deserializer", 
> "org.apache.kafka.common.serialization.ByteArrayDeserializer")
>   props.put("value.deserializer", 
> "org.apache.kafka.common.serialization.ByteArrayDeserializer")
>   val consumer = new KafkaConsumer[Array[Byte], Array[Byte]](props)
>   import scala.collection.JavaConversions._
>   def dumpPositionAndCommitted() = {
> consumer.assignment().foreach { tp =>
>   println(tp)
>   println(s"Position - ${consumer.position(tp)}")
>   println(s"Committed - ${consumer.committed(tp)}")
> }
> println("---")
>   }
>   consumer.assign(Collections.singleton(new TopicPartition("topic", 0)))
>   dumpPositionAndCommitted()
>   Thread.sleep(3000)
>   val ps = Collections.singleton(new TopicPartition("topic", 1)) ++ 
> consumer.assignment()
>   consumer.assign(ps)
>   dumpPositionAndCommitted()
>   Thread.sleep(3000)
>   dumpPositionAndCommitted()
> {code}
> and the result is
> {noformat}
> Position - 1211046445
> Committed - OffsetAndMetadata{offset=1211046445, metadata=''}
> ---
> proto7_fraud-1
> Position - 1262864347
> Committed - null
> topic-0
> Position - 1211046445
> Committed - OffsetAndMetadata{offset=1211046445, metadata=''}
> ---
> topic-1
> Position - 1262864347
> Committed - null
> topic-0
> Position - 1211046445
> Committed - OffsetAndMetadata{offset=1211046445, metadata=''}
> ---
> {noformat}
> Pay attention to 
> {noformat}
> topic-1
> Position - 1262864347
> Committed - null
> {noformat}
> There is no committed offset fetched from broker, but it is. Looks like we 
> should set {{needsFetchCommittedOffsets}} to {{true}} during assign the 
> partition



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-66 Kafka Connect Transformers for messages

[Nisarg Shah]

Thank you for your inputs Gwen and Michael.

The original reason why I suggested a set based processing is because of the 
flexibility is provides. The JIRA had a comment by a user requesting a feature 
that could be achieved with this.

After reading Gwen and Michael's points, (I went through the documentation and 
the code in detail) and agree with what you have to say. Also, fewer guarantees 
make what I had in mind less certain and thus simplifying it to a single 
message based transformation would ensure that users who do require more 
flexibility with the transformations will automatically “turn to" Kafka 
Streams. The transformation logic on a message by message basis makes more 
sense.

One usecase that Kafka Connect could consider is adding or removing a message 
completely. (This was trivially possible with the collection passing). Should 
users be pointed towards Kafka Streams even for this use case? I think this is 
a very useful feature for Connect too, and I’ll try to rethink on the API too.

Removing a message is as easy as returning a null and having the next 
transformer skip it, but adding messages would involve say a queue between 
transformers and a method which says “pass message” to the next, which can be 
called multiple times from one “transform” function; a variation on the chain 
of responsibility design pattern.

> On Jul 12, 2016, at 12:54 AM, Michael Noll  wrote:
> 
> As Gwen said, my initial thought is that message transformations that are
> "more than trivial" should rather be done by Kafka Streams, rather than by
> Kafka Connect (for the reasons that Gwen mentioned).
> 
> Transforming one message at a time would be a good fit for Kafka Connect.
> An important use case is to remove sensitive data (such as PII) from an
> incoming data stream before it hits Kafka's persistent storage -- this use
> case can't be implemented well with Kafka Streams because, by design, Kafka
> Streams is meant to read its input data from Kafka (i.e. at the point when
> Kafka Streams could be used to removed sensitive data fields the data is
> already stored persistently in Kafka, and this might be a no-go depending
> on the use case).
> 
> I'm of course interested to hear what other people think.
> 
> 
> On Tue, Jul 12, 2016 at 6:06 AM, Gwen Shapira  wrote:
> 
>> I think we need to restrict the functionality to one-message-at-a-time.
>> 
>> Basically, connect gives very little guarantees about the size of the set
>> of the composition (you may get same messages over and over, mix of old and
>> new, etc)
>> 
>> In order to do useful things over a collection, you need better defined
>> semantics of what's included. Kafka Streams is putting tons of effort into
>> having good windowing semantics, and I think apps that require modification
>> of collections are a better fit there.
>> 
>> I'm willing to change my mind though (have been known to happen) - what are
>> the comments about usage that point toward the collections approach?
>> 
>> Gwen
>> 
>> On Mon, Jul 11, 2016 at 3:32 PM, Nisarg Shah  wrote:
>> 
>>> Thanks Jay, added that to the KIP.
>>> 
>>> Besides reviewing the KIP as a whole, I wanted to know about what
>> everyone
>>> thinks about what data should be dealt at the Transformer level.
>> Transform
>>> the whole Collection of Records (giving the flexibility of modifying
>>> messages across the set) OR
>>> Transform messages one at a time, iteratively. This will restrict
>>> modifications across messages.
>>> 
>>> I’ll get a working sample ready soon, to have a look. There were some
>>> comments about Transformer usage that pointed to the first approach,
>> which
>>> I prefer too given the flexibility.
>>> 
 On Jul 11, 2016, at 2:49 PM, Jay Kreps  wrote:
 
 One minor thing, the Transformer interface probably needs a close()
>>> method
 (i.e. the opposite of initialize). This would be used for any
>> transformer
 that uses a resource like a file/socket/db connection/etc that needs to
>>> be
 closed. You usually don't need this but when you do need it you really
>>> need
 it.
 
 -Jay
 
 On Mon, Jul 11, 2016 at 1:47 PM, Nisarg Shah 
>> wrote:
 
> Hello,
> 
> This KIP <
> 
>>> 
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-66:+Add+Kafka+Connect+Transformers+to+allow+transformations+to+messages
 
> is for KAFKA-3209 .
> It’s about capabilities to transform messages in Kafka Connect.
> 
> Some design decisions need to be taken, so please advise me on the
>> same.
> Feel free to express any thoughts or concerns as well.
> 
> Many many thanks to Ewen Cheslack-Postava.
> 
> -Nisarg
>>> 
>>> 
>> 
> 
> 
> 
> -- 
> Best regards,
> Michael Noll
> 
> 
> 
> *Michael G. Noll | Product Manager | Confluent | +1 650.453.5860Download
> Apache Kafka 

[jira] [Commented] (KAFKA-2431) Test SSL/TLS impact on performance

[Jiangjie Qin (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15378759#comment-15378759
 ] 

Jiangjie Qin commented on KAFKA-2431:
-

[~benstopford] It seems I am not able to access the Google doc anymore. We 
recently did some performance test of SSL at LinkedIn and saw some interesting 
results. We would like to compare that with the tests done previously. Do you 
mind letting us check on the previous result? Thanks.

> Test SSL/TLS impact on performance
> --
>
> Key: KAFKA-2431
> URL: https://issues.apache.org/jira/browse/KAFKA-2431
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Ismael Juma
>Assignee: Ben Stopford
>Priority: Blocker
> Fix For: 0.9.0.0
>
>
> Test new Producer and new Consumer performance with and without SSL/TLS once 
> the SSL/TLS branch is integrated.
> The ideal scenario is that SSL/TLS would not have an impact if disabled. When 
> enabled, there will be some overhead (encryption and the inability to use 
> `SendFile`) and it will be good to quantify it. The encryption overhead is 
> reduced if recent JDKs are used with CPUs that support AES-specific 
> instructions (https://en.wikipedia.org/wiki/AES_instruction_set).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder

[Bharat Viswanadham (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15378617#comment-15378617
 ] 

Bharat Viswanadham commented on KAFKA-3729:
---

[~thoughtp...@gmail.com][~guozhang] Can you provide some additional 
information, as i am new to streams, it will help me to start work on this 
JIRA. 

Thanks,
Bharat


>  Auto-configure non-default SerDes passed alongside the topology builder
> 
>
> Key: KAFKA-3729
> URL: https://issues.apache.org/jira/browse/KAFKA-3729
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Fred Patton
>Assignee: Bharat Viswanadham
>  Labels: api, newbie
>
> From Guozhang Wang:
> "Only default serdes provided through configs are auto-configured today. But 
> we could auto-configure other serdes passed alongside the topology builder as 
> well."



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-3948) Invalid broker port in Zookeeper when SSL is enabled

[Bharat Viswanadham (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bharat Viswanadham resolved KAFKA-3948.
---
Resolution: Not A Problem

> Invalid broker port in Zookeeper when SSL is enabled
> 
>
> Key: KAFKA-3948
> URL: https://issues.apache.org/jira/browse/KAFKA-3948
> Project: Kafka
>  Issue Type: Bug
>  Components: core, network
>Affects Versions: 0.9.0.1
>Reporter: Gérald Quintana
>Assignee: Bharat Viswanadham
>
> With broker config
> {code}
> listeners=SSL://:9093,PLAINTEXT://:9092
> port=9093
> {code}
> gives in Zookeeper /brokers/ids/1
> {code}
> {"jmx_port":,"timestamp":"1468249905473","endpoints":["SSL://kafka1:9093","PLAINTEXT://kafka1:9092"],"host":"kafka1","version":2,"port":9092}
> {code}
> Notice that port 9092 not 9093
> Then, different scenario, with config:
> {code}
> listeners=SSL://:9093
> port=9093
> {code}
> gives in Zookeeper /brokers/ids/1
> {code}
> {"jmx_port":,"timestamp":"1468250372974","endpoints":["SSL://kafka1:9093"],"host":null,"version":2,"port":-1}
> {code}
> Now host is null and port is -1
> Setting advertised.port doesn't help



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3948) Invalid broker port in Zookeeper when SSL is enabled

[Bharat Viswanadham (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15378604#comment-15378604
 ] 

Bharat Viswanadham commented on KAFKA-3948:
---

yes got that point. Thank you for info. [~sriharsha]

> Invalid broker port in Zookeeper when SSL is enabled
> 
>
> Key: KAFKA-3948
> URL: https://issues.apache.org/jira/browse/KAFKA-3948
> Project: Kafka
>  Issue Type: Bug
>  Components: core, network
>Affects Versions: 0.9.0.1
>Reporter: Gérald Quintana
>Assignee: Bharat Viswanadham
>
> With broker config
> {code}
> listeners=SSL://:9093,PLAINTEXT://:9092
> port=9093
> {code}
> gives in Zookeeper /brokers/ids/1
> {code}
> {"jmx_port":,"timestamp":"1468249905473","endpoints":["SSL://kafka1:9093","PLAINTEXT://kafka1:9092"],"host":"kafka1","version":2,"port":9092}
> {code}
> Notice that port 9092 not 9093
> Then, different scenario, with config:
> {code}
> listeners=SSL://:9093
> port=9093
> {code}
> gives in Zookeeper /brokers/ids/1
> {code}
> {"jmx_port":,"timestamp":"1468250372974","endpoints":["SSL://kafka1:9093"],"host":null,"version":2,"port":-1}
> {code}
> Now host is null and port is -1
> Setting advertised.port doesn't help



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3948) Invalid broker port in Zookeeper when SSL is enabled

[Sriharsha Chintalapani (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15378603#comment-15378603
 ] 

Sriharsha Chintalapani commented on KAFKA-3948:
---

[~bharatviswa] [~gquintana] host and port in zookeeper data is used to maintain 
the backward compatibility. So only plaintext port and host will be registered 
there. If you are depending zookeeper data to parse broker hosts I suggest you 
to look under endpoints section.


> Invalid broker port in Zookeeper when SSL is enabled
> 
>
> Key: KAFKA-3948
> URL: https://issues.apache.org/jira/browse/KAFKA-3948
> Project: Kafka
>  Issue Type: Bug
>  Components: core, network
>Affects Versions: 0.9.0.1
>Reporter: Gérald Quintana
>Assignee: Bharat Viswanadham
>
> With broker config
> {code}
> listeners=SSL://:9093,PLAINTEXT://:9092
> port=9093
> {code}
> gives in Zookeeper /brokers/ids/1
> {code}
> {"jmx_port":,"timestamp":"1468249905473","endpoints":["SSL://kafka1:9093","PLAINTEXT://kafka1:9092"],"host":"kafka1","version":2,"port":9092}
> {code}
> Notice that port 9092 not 9093
> Then, different scenario, with config:
> {code}
> listeners=SSL://:9093
> port=9093
> {code}
> gives in Zookeeper /brokers/ids/1
> {code}
> {"jmx_port":,"timestamp":"1468250372974","endpoints":["SSL://kafka1:9093"],"host":null,"version":2,"port":-1}
> {code}
> Now host is null and port is -1
> Setting advertised.port doesn't help



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3173) Error while moving some partitions to OnlinePartition state

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3173:
---
Fix Version/s: (was: 0.10.0.1)
   0.10.1.0

> Error while moving some partitions to OnlinePartition state 
> 
>
> Key: KAFKA-3173
> URL: https://issues.apache.org/jira/browse/KAFKA-3173
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.0
>Reporter: Flavio Junqueira
>Assignee: Flavio Junqueira
>Priority: Critical
> Fix For: 0.10.1.0
>
> Attachments: KAFKA-3173-race-repro.patch
>
>
> We observed another instance of the problem reported in KAFKA-2300, but this 
> time the error appeared in the partition state machine. In KAFKA-2300, we 
> haven't cleaned up the state in {{PartitionStateMachine}} and 
> {{ReplicaStateMachine}} as we do in {{KafkaController}}.
> Here is the stack trace:
> {noformat}
> 2016-01-29 15:26:51,393] ERROR [Partition state machine on Controller 0]: 
> Error while moving some partitions to OnlinePartition state 
> (kafka.controller.PartitionStateMachine)java.lang.IllegalStateException: 
> Controller to broker state change requests batch is not empty while creating 
> a new one. 
> Some LeaderAndIsr state changes Map(0 -> Map(foo-0 -> (LeaderAndIsrInfo:
> (Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1),ReplicationFactor:1),AllReplicas:0)))
>  might be lostat 
> kafka.controller.ControllerBrokerRequestBatch.newBatch(ControllerChannelManager.scala:254)
> at 
> kafka.controller.PartitionStateMachine.handleStateChanges(PartitionStateMachine.scala:144)
> at 
> kafka.controller.KafkaController.onNewPartitionCreation(KafkaController.scala:517)
> at 
> kafka.controller.KafkaController.onNewTopicCreation(KafkaController.scala:504)
> at 
> kafka.controller.PartitionStateMachine$TopicChangeListener$$anonfun$handleChildChange$1.apply$mcV$sp(PartitionStateMachine.scala:437)
> at 
> kafka.controller.PartitionStateMachine$TopicChangeListener$$anonfun$handleChildChange$1.apply(PartitionStateMachine.scala:419)
> at 
> kafka.controller.PartitionStateMachine$TopicChangeListener$$anonfun$handleChildChange$1.apply(PartitionStateMachine.scala:419)
> at 
> kafka.utils.CoreUtils$.inLock(CoreUtils.scala:262)at 
> kafka.controller.PartitionStateMachine$TopicChangeListener.handleChildChange(PartitionStateMachine.scala:418)
> at 
> org.I0Itec.zkclient.ZkClient$10.run(ZkClient.java:842)at 
> org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1194) The kafka broker cannot delete the old log files after the configured time

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15378237#comment-15378237
 ] 

ASF GitHub Bot commented on KAFKA-1194:
---

GitHub user rperi opened a pull request:

https://github.com/apache/kafka/pull/1624

KAFKA-1194: New config to specific memory mapped file update capability of 
underlying OS

A new configuration value that can be used to indicate if the underlying OS 
(and environment) supports updates to memory mapped files. 

On windows systems, metadata updates, and rename operations fail if the 
file is still memory mapped. 
This fix will close the file if updates are not supported, and then 
proceeds to execute the update operations.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rperi/kafka trunk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1624.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1624


commit 779b5d0eb101cd4c2373acaa88441533af6335d6
Author: rperi 
Date:   2016-07-14T19:00:28Z

Introduces new configuration to specify if underlying OS or environment 
supports updates to memory mapped files




> The kafka broker cannot delete the old log files after the configured time
> --
>
> Key: KAFKA-1194
> URL: https://issues.apache.org/jira/browse/KAFKA-1194
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.8.1
> Environment: window
>Reporter: Tao Qin
>Assignee: Jay Kreps
>  Labels: features, patch
> Fix For: 0.10.1.0
>
> Attachments: KAFKA-1194.patch, kafka-1194-v1.patch, 
> kafka-1194-v2.patch
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> We tested it in windows environment, and set the log.retention.hours to 24 
> hours.
> # The minimum age of a log file to be eligible for deletion
> log.retention.hours=24
> After several days, the kafka broker still cannot delete the old log file. 
> And we get the following exceptions:
> [2013-12-19 01:57:38,528] ERROR Uncaught exception in scheduled task 
> 'kafka-log-retention' (kafka.utils.KafkaScheduler)
> kafka.common.KafkaStorageException: Failed to change the log file suffix from 
>  to .deleted for log segment 1516723
>  at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:249)
>  at kafka.log.Log.kafka$log$Log$$asyncDeleteSegment(Log.scala:638)
>  at kafka.log.Log.kafka$log$Log$$deleteSegment(Log.scala:629)
>  at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:418)
>  at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:418)
>  at 
> scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
>  at scala.collection.immutable.List.foreach(List.scala:76)
>  at kafka.log.Log.deleteOldSegments(Log.scala:418)
>  at 
> kafka.log.LogManager.kafka$log$LogManager$$cleanupExpiredSegments(LogManager.scala:284)
>  at 
> kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:316)
>  at 
> kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:314)
>  at 
> scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:743)
>  at scala.collection.Iterator$class.foreach(Iterator.scala:772)
>  at 
> scala.collection.JavaConversions$JIteratorWrapper.foreach(JavaConversions.scala:573)
>  at scala.collection.IterableLike$class.foreach(IterableLike.scala:73)
>  at 
> scala.collection.JavaConversions$JListWrapper.foreach(JavaConversions.scala:615)
>  at 
> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:742)
>  at kafka.log.LogManager.cleanupLogs(LogManager.scala:314)
>  at 
> kafka.log.LogManager$$anonfun$startup$1.apply$mcV$sp(LogManager.scala:143)
>  at kafka.utils.KafkaScheduler$$anon$1.run(KafkaScheduler.scala:100)
>  at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>  at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>  at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  at 

[GitHub] kafka pull request #1624: KAFKA-1194: New config to specific memory mapped f...

[rperi]

GitHub user rperi opened a pull request:

https://github.com/apache/kafka/pull/1624

KAFKA-1194: New config to specific memory mapped file update capability of 
underlying OS

A new configuration value that can be used to indicate if the underlying OS 
(and environment) supports updates to memory mapped files. 

On windows systems, metadata updates, and rename operations fail if the 
file is still memory mapped. 
This fix will close the file if updates are not supported, and then 
proceeds to execute the update operations.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rperi/kafka trunk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1624.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1624


commit 779b5d0eb101cd4c2373acaa88441533af6335d6
Author: rperi 
Date:   2016-07-14T19:00:28Z

Introduces new configuration to specify if underlying OS or environment 
supports updates to memory mapped files




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-3966) KafkaConsumer briefly ignores partitions on backlogs

[Kanak Biscuitwala (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kanak Biscuitwala updated KAFKA-3966:
-
Summary: KafkaConsumer briefly ignores partitions on backlogs  (was: 
Consumer briefly ignores partitions on backlogs)

> KafkaConsumer briefly ignores partitions on backlogs
> 
>
> Key: KAFKA-3966
> URL: https://issues.apache.org/jira/browse/KAFKA-3966
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.10.0.0
>Reporter: Kanak Biscuitwala
> Attachments: screenshot-1.png
>
>
> Setup:
> Kafka 0.10.0.0
> 9 brokers
> 36 partitions
> 12 members in the consumer group
> 5 producers, partitioning data roughly evenly
> max.poll.records = 10
> ~1500 events/sec
> ~500 bytes/message
> KafkaConsumer API
> In the following setup, 3 partitions are assigned to each consumer (and 12 
> are served by each broker). I've noticed that the third of the three 
> partitions tends to be ignored in favor of the first two on each consumer, 
> even though the third partition has data available. Only until the first two 
> partitions are fully caught up does the consumer return back messages from 
> the third. This causes a frustrating imbalance in which the same partitions 
> always fall behind.
> As a side note, this is true for all of our topics, regardless of 
> partitioning strategy. The problem goes away if there are exactly as many 
> consumers as partitions.
> I can attach a screenshot showing the same partitions falling behind 
> (verified that they're each assigned to different nodes), if that is helpful.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3966) Consumer briefly ignores partitions on backlogs

[Kanak Biscuitwala (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kanak Biscuitwala updated KAFKA-3966:
-
Affects Version/s: 0.10.0.0

> Consumer briefly ignores partitions on backlogs
> ---
>
> Key: KAFKA-3966
> URL: https://issues.apache.org/jira/browse/KAFKA-3966
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.10.0.0
>Reporter: Kanak Biscuitwala
> Attachments: screenshot-1.png
>
>
> Setup:
> Kafka 0.10.0.0
> 9 brokers
> 36 partitions
> 12 members in the consumer group
> 5 producers, partitioning data roughly evenly
> max.poll.records = 10
> ~1500 events/sec
> ~500 bytes/message
> KafkaConsumer API
> In the following setup, 3 partitions are assigned to each consumer (and 12 
> are served by each broker). I've noticed that the third of the three 
> partitions tends to be ignored in favor of the first two on each consumer, 
> even though the third partition has data available. Only until the first two 
> partitions are fully caught up does the consumer return back messages from 
> the third. This causes a frustrating imbalance in which the same partitions 
> always fall behind.
> As a side note, this is true for all of our topics, regardless of 
> partitioning strategy. The problem goes away if there are exactly as many 
> consumers as partitions.
> I can attach a screenshot showing the same partitions falling behind 
> (verified that they're each assigned to different nodes), if that is helpful.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3966) Consumer briefly ignores partitions on backlogs

[Kanak Biscuitwala (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kanak Biscuitwala updated KAFKA-3966:
-
Component/s: consumer

> Consumer briefly ignores partitions on backlogs
> ---
>
> Key: KAFKA-3966
> URL: https://issues.apache.org/jira/browse/KAFKA-3966
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.10.0.0
>Reporter: Kanak Biscuitwala
> Attachments: screenshot-1.png
>
>
> Setup:
> Kafka 0.10.0.0
> 9 brokers
> 36 partitions
> 12 members in the consumer group
> 5 producers, partitioning data roughly evenly
> max.poll.records = 10
> ~1500 events/sec
> ~500 bytes/message
> KafkaConsumer API
> In the following setup, 3 partitions are assigned to each consumer (and 12 
> are served by each broker). I've noticed that the third of the three 
> partitions tends to be ignored in favor of the first two on each consumer, 
> even though the third partition has data available. Only until the first two 
> partitions are fully caught up does the consumer return back messages from 
> the third. This causes a frustrating imbalance in which the same partitions 
> always fall behind.
> As a side note, this is true for all of our topics, regardless of 
> partitioning strategy. The problem goes away if there are exactly as many 
> consumers as partitions.
> I can attach a screenshot showing the same partitions falling behind 
> (verified that they're each assigned to different nodes), if that is helpful.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3966) Consumer briefly ignores partitions on backlogs

[Kanak Biscuitwala (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kanak Biscuitwala updated KAFKA-3966:
-
Attachment: screenshot-1.png

> Consumer briefly ignores partitions on backlogs
> ---
>
> Key: KAFKA-3966
> URL: https://issues.apache.org/jira/browse/KAFKA-3966
> Project: Kafka
>  Issue Type: Bug
>Reporter: Kanak Biscuitwala
> Attachments: screenshot-1.png
>
>
> Setup:
> Kafka 0.10.0.0
> 9 brokers
> 36 partitions
> 12 members in the consumer group
> 5 producers, partitioning data roughly evenly
> max.poll.records = 10
> ~1500 events/sec
> ~500 bytes/message
> KafkaConsumer API
> In the following setup, 3 partitions are assigned to each consumer (and 12 
> are served by each broker). I've noticed that the third of the three 
> partitions tends to be ignored in favor of the first two on each consumer, 
> even though the third partition has data available. Only until the first two 
> partitions are fully caught up does the consumer return back messages from 
> the third. This causes a frustrating imbalance in which the same partitions 
> always fall behind.
> As a side note, this is true for all of our topics, regardless of 
> partitioning strategy. The problem goes away if there are exactly as many 
> consumers as partitions.
> I can attach a screenshot showing the same partitions falling behind 
> (verified that they're each assigned to different nodes), if that is helpful.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-3966) Consumer briefly ignores partitions on backlogs

[Kanak Biscuitwala (JIRA)]

Kanak Biscuitwala created KAFKA-3966:


 Summary: Consumer briefly ignores partitions on backlogs
 Key: KAFKA-3966
 URL: https://issues.apache.org/jira/browse/KAFKA-3966
 Project: Kafka
  Issue Type: Bug
Reporter: Kanak Biscuitwala
 Attachments: screenshot-1.png

Setup:
Kafka 0.10.0.0
9 brokers
36 partitions
12 members in the consumer group
5 producers, partitioning data roughly evenly
max.poll.records = 10
~1500 events/sec
~500 bytes/message
KafkaConsumer API

In the following setup, 3 partitions are assigned to each consumer (and 12 are 
served by each broker). I've noticed that the third of the three partitions 
tends to be ignored in favor of the first two on each consumer, even though the 
third partition has data available. Only until the first two partitions are 
fully caught up does the consumer return back messages from the third. This 
causes a frustrating imbalance in which the same partitions always fall behind.

As a side note, this is true for all of our topics, regardless of partitioning 
strategy. The problem goes away if there are exactly as many consumers as 
partitions.

I can attach a screenshot showing the same partitions falling behind (verified 
that they're each assigned to different nodes), if that is helpful.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3940) Log should check the return value of dir.mkdirs()

[Jim Jagielski (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15378180#comment-15378180
 ] 

Jim Jagielski commented on KAFKA-3940:
--

In some places we do a dir.mkdirs() and check the return, and throw an error if 
it fails.

Assuming we want to standardize, should we continue w/ that format or should we 
convert all to using Files.createDirectory() (and a general IOException)? I 
don't think we should mix and match :) 

> Log should check the return value of dir.mkdirs()
> -
>
> Key: KAFKA-3940
> URL: https://issues.apache.org/jira/browse/KAFKA-3940
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.10.0.0
>Reporter: Jun Rao
>Assignee: Ishita Mandhan
>  Labels: newbie
>
> In Log.loadSegments(), we call dir.mkdirs() w/o checking the return value and 
> just assume the directory will exist after the call. However, if the 
> directory can't be created (e.g. due to no space), we will hit 
> NullPointerException in the next statement, which will be confusing.
>for(file <- dir.listFiles if file.isFile) {



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3940) Log should check the return value of dir.mkdirs()

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15378157#comment-15378157
 ] 

ASF GitHub Bot commented on KAFKA-3940:
---

Github user jimjag closed the pull request at:

https://github.com/apache/kafka/pull/1620


> Log should check the return value of dir.mkdirs()
> -
>
> Key: KAFKA-3940
> URL: https://issues.apache.org/jira/browse/KAFKA-3940
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.10.0.0
>Reporter: Jun Rao
>Assignee: Ishita Mandhan
>  Labels: newbie
>
> In Log.loadSegments(), we call dir.mkdirs() w/o checking the return value and 
> just assume the directory will exist after the call. However, if the 
> directory can't be created (e.g. due to no space), we will hit 
> NullPointerException in the next statement, which will be confusing.
>for(file <- dir.listFiles if file.isFile) {



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1620: KAFKA-3940: Log should check the return value of d...

[jimjag]

Github user jimjag closed the pull request at:

https://github.com/apache/kafka/pull/1620


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3940) Log should check the return value of dir.mkdirs()

[Jim Jagielski (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15378153#comment-15378153
 ] 

Jim Jagielski commented on KAFKA-3940:
--

I'm fine w/ working w/ Ishita... What I was planning is handling the specific 
problem set and then doing an generalized refactor.

> Log should check the return value of dir.mkdirs()
> -
>
> Key: KAFKA-3940
> URL: https://issues.apache.org/jira/browse/KAFKA-3940
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.10.0.0
>Reporter: Jun Rao
>Assignee: Ishita Mandhan
>  Labels: newbie
>
> In Log.loadSegments(), we call dir.mkdirs() w/o checking the return value and 
> just assume the directory will exist after the call. However, if the 
> directory can't be created (e.g. due to no space), we will hit 
> NullPointerException in the next statement, which will be confusing.
>for(file <- dir.listFiles if file.isFile) {



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3948) Invalid broker port in Zookeeper when SSL is enabled

[Bharat Viswanadham (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15378099#comment-15378099
 ] 

Bharat Viswanadham commented on KAFKA-3948:
---

You can also refer the code for confirmation
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/KafkaHealthcheck.scala
Refer lines 66-72.

> Invalid broker port in Zookeeper when SSL is enabled
> 
>
> Key: KAFKA-3948
> URL: https://issues.apache.org/jira/browse/KAFKA-3948
> Project: Kafka
>  Issue Type: Bug
>  Components: core, network
>Affects Versions: 0.9.0.1
>Reporter: Gérald Quintana
>Assignee: Bharat Viswanadham
>
> With broker config
> {code}
> listeners=SSL://:9093,PLAINTEXT://:9092
> port=9093
> {code}
> gives in Zookeeper /brokers/ids/1
> {code}
> {"jmx_port":,"timestamp":"1468249905473","endpoints":["SSL://kafka1:9093","PLAINTEXT://kafka1:9092"],"host":"kafka1","version":2,"port":9092}
> {code}
> Notice that port 9092 not 9093
> Then, different scenario, with config:
> {code}
> listeners=SSL://:9093
> port=9093
> {code}
> gives in Zookeeper /brokers/ids/1
> {code}
> {"jmx_port":,"timestamp":"1468250372974","endpoints":["SSL://kafka1:9093"],"host":null,"version":2,"port":-1}
> {code}
> Now host is null and port is -1
> Setting advertised.port doesn't help



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3948) Invalid broker port in Zookeeper when SSL is enabled

[Bharat Viswanadham (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bharat Viswanadham updated KAFKA-3948:
--
Component/s: core

> Invalid broker port in Zookeeper when SSL is enabled
> 
>
> Key: KAFKA-3948
> URL: https://issues.apache.org/jira/browse/KAFKA-3948
> Project: Kafka
>  Issue Type: Bug
>  Components: core, network
>Affects Versions: 0.9.0.1
>Reporter: Gérald Quintana
>Assignee: Bharat Viswanadham
>
> With broker config
> {code}
> listeners=SSL://:9093,PLAINTEXT://:9092
> port=9093
> {code}
> gives in Zookeeper /brokers/ids/1
> {code}
> {"jmx_port":,"timestamp":"1468249905473","endpoints":["SSL://kafka1:9093","PLAINTEXT://kafka1:9092"],"host":"kafka1","version":2,"port":9092}
> {code}
> Notice that port 9092 not 9093
> Then, different scenario, with config:
> {code}
> listeners=SSL://:9093
> port=9093
> {code}
> gives in Zookeeper /brokers/ids/1
> {code}
> {"jmx_port":,"timestamp":"1468250372974","endpoints":["SSL://kafka1:9093"],"host":null,"version":2,"port":-1}
> {code}
> Now host is null and port is -1
> Setting advertised.port doesn't help



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3948) Invalid broker port in Zookeeper when SSL is enabled

[Bharat Viswanadham (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15377908#comment-15377908
 ] 

Bharat Viswanadham commented on KAFKA-3948:
---

Pls let me know your comments on my inputs.

> Invalid broker port in Zookeeper when SSL is enabled
> 
>
> Key: KAFKA-3948
> URL: https://issues.apache.org/jira/browse/KAFKA-3948
> Project: Kafka
>  Issue Type: Bug
>  Components: network
>Affects Versions: 0.9.0.1
>Reporter: Gérald Quintana
>Assignee: Bharat Viswanadham
>
> With broker config
> {code}
> listeners=SSL://:9093,PLAINTEXT://:9092
> port=9093
> {code}
> gives in Zookeeper /brokers/ids/1
> {code}
> {"jmx_port":,"timestamp":"1468249905473","endpoints":["SSL://kafka1:9093","PLAINTEXT://kafka1:9092"],"host":"kafka1","version":2,"port":9092}
> {code}
> Notice that port 9092 not 9093
> Then, different scenario, with config:
> {code}
> listeners=SSL://:9093
> port=9093
> {code}
> gives in Zookeeper /brokers/ids/1
> {code}
> {"jmx_port":,"timestamp":"1468250372974","endpoints":["SSL://kafka1:9093"],"host":null,"version":2,"port":-1}
> {code}
> Now host is null and port is -1
> Setting advertised.port doesn't help



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-3948) Invalid broker port in Zookeeper when SSL is enabled

[Bharat Viswanadham (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15377904#comment-15377904
 ] 

Bharat Viswanadham edited comment on KAFKA-3948 at 7/14/16 6:16 PM:


Hi, 
The behavior you are observing is correct.
Host - Default host
Port - Default Port

The host and port are the for compatibility with old client.
The host and port are updated if we have listener with PLAINTEXT. As PLAINTEXT 
is supported as default.

In your first scenario you have both PLAINTEXT and SSL 
(listeners=SSL://:9093,PLAINTEXT://:9092
port=9093)
so, the host got updated to kafka1 (your kafka cluster host)
and port 9093 (PLAINTEXT port)

In the second scenario, you have only SSL (listeners=SSL://:9093)
then the host and port are set to default values 
for host it is null
port it is -1




was (Author: bharatviswa):
Hi, 
The behavior you are observing is correct.
Host - Default host
Port - Default Port

The host and port are the for compatibility with old client.
The host and port are updated if we have listener with PLAINTEXT. As PLAINTEXT 
is supported as default.

In your first scenario you have both PLAINTEXT and SSL 
(listeners=SSL://:9093,PLAINTEXT://:9092
port=9093)
so, the host got updated to kafka1 (your kafka cluster host)
and port 9093 (PLAINTEXT port)

In the second scenario, you have onlt SSL (listeners=SSL://:9093)
then the host and port are set to default values 
for host it is null
port it is -1



> Invalid broker port in Zookeeper when SSL is enabled
> 
>
> Key: KAFKA-3948
> URL: https://issues.apache.org/jira/browse/KAFKA-3948
> Project: Kafka
>  Issue Type: Bug
>  Components: network
>Affects Versions: 0.9.0.1
>Reporter: Gérald Quintana
>Assignee: Bharat Viswanadham
>
> With broker config
> {code}
> listeners=SSL://:9093,PLAINTEXT://:9092
> port=9093
> {code}
> gives in Zookeeper /brokers/ids/1
> {code}
> {"jmx_port":,"timestamp":"1468249905473","endpoints":["SSL://kafka1:9093","PLAINTEXT://kafka1:9092"],"host":"kafka1","version":2,"port":9092}
> {code}
> Notice that port 9092 not 9093
> Then, different scenario, with config:
> {code}
> listeners=SSL://:9093
> port=9093
> {code}
> gives in Zookeeper /brokers/ids/1
> {code}
> {"jmx_port":,"timestamp":"1468250372974","endpoints":["SSL://kafka1:9093"],"host":null,"version":2,"port":-1}
> {code}
> Now host is null and port is -1
> Setting advertised.port doesn't help



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3948) Invalid broker port in Zookeeper when SSL is enabled

[Bharat Viswanadham (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15377904#comment-15377904
 ] 

Bharat Viswanadham commented on KAFKA-3948:
---

Hi, 
The behavior you are observing is correct.
Host - Default host
Port - Default Port

The host and port are the for compatibility with old client.
The host and port are updated if we have listener with PLAINTEXT. As PLAINTEXT 
is supported as default.

In your first scenario you have both PLAINTEXT and SSL 
(listeners=SSL://:9093,PLAINTEXT://:9092
port=9093)
so, the host got updated to kafka1 (your kafka cluster host)
and port 9093 (PLAINTEXT port)

In the second scenario, you have onlt SSL (listeners=SSL://:9093)
then the host and port are set to default values 
for host it is null
port it is -1



> Invalid broker port in Zookeeper when SSL is enabled
> 
>
> Key: KAFKA-3948
> URL: https://issues.apache.org/jira/browse/KAFKA-3948
> Project: Kafka
>  Issue Type: Bug
>  Components: network
>Affects Versions: 0.9.0.1
>Reporter: Gérald Quintana
>Assignee: Bharat Viswanadham
>
> With broker config
> {code}
> listeners=SSL://:9093,PLAINTEXT://:9092
> port=9093
> {code}
> gives in Zookeeper /brokers/ids/1
> {code}
> {"jmx_port":,"timestamp":"1468249905473","endpoints":["SSL://kafka1:9093","PLAINTEXT://kafka1:9092"],"host":"kafka1","version":2,"port":9092}
> {code}
> Notice that port 9092 not 9093
> Then, different scenario, with config:
> {code}
> listeners=SSL://:9093
> port=9093
> {code}
> gives in Zookeeper /brokers/ids/1
> {code}
> {"jmx_port":,"timestamp":"1468250372974","endpoints":["SSL://kafka1:9093"],"host":null,"version":2,"port":-1}
> {code}
> Now host is null and port is -1
> Setting advertised.port doesn't help



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3859) Consumer group is stuck in rebalancing status

[Vahid Hashemian (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vahid Hashemian updated KAFKA-3859:
---
Status: Patch Available  (was: In Progress)

> Consumer group is stuck in rebalancing status
> -
>
> Key: KAFKA-3859
> URL: https://issues.apache.org/jira/browse/KAFKA-3859
> Project: Kafka
>  Issue Type: Bug
>Reporter: Vahid Hashemian
>Assignee: Vahid Hashemian
>
> * I have a topic (1 partition) and a producer and new consumer that produce 
> to and consumer from the topic.
> * The consumer belongs to group {{A}}.
> * I kill the consumer (whether it has consumed any messages or not does not 
> seem to be relevant).
> * After a short period when group status is processed and finalized, I run 
> the consumer-group describe command ({{kafka-consumer-groups.sh 
> --bootstrap-server localhost:9092 --new-consumer --describe --group A}}).
> * The response I receive is {{Consumer group `A` is rebalancing.}}
> * I keep trying the command but the response does not change.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[DISCUSS] KIP-4 ACL Admin Schema

[Grant Henke]

The KIP-4 Delete Topic Schema vote has passed and the patch
 is available for review. Now I
would like to start the discussion for the Acls request/response and server
side implementations. This includes the ListAclsRequest/Response and the
AlterAclsRequest/Response.

Details for this implementation can be read here:
*https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-ACLAdminSchema(KAFKA-3266)
*

I have included the exact content below for clarity:

> ACL Admin Schema (KAFKA-3266
> )
>
> *Note*: Some of this work/code overlaps with "KIP-50 - Move Authorizer to
> o.a.k.common package
> ".
> KIP-4 does not change the Authorizer interface at all, but does provide
> java objects in "org.apache.kafka.common.security.auth" to be used in the
> protocol request/response classes. It also provides translations between
> the Java and Scala versions for server side compatibility with
> the Authorizer interface.
>
> List ACLs Request
>
>
>
> ListAcls Request (Version: 0) => principal resource
>   principal => NULLABLE_STRING
>   resource => resource_type resource_name
> resource_type => INT8
> resource_name => STRING
>
> Request semantics:
>
>1. Can be sent to any broker
>2. If a non-null principal is provided the returned ACLs will be
>filtered by that principle, otherwise ACLs for all principals will be
>listed.
>3. If a resource with a resource_type != -1 is provided ACLs will be
>filtered by that resource, otherwise ACLs for all resources will be listed.
>4. Any principle can list their own ACLs where the permission type is
>"Allow", Otherwise the principle must be authorized to the "All" Operation
>on the "Cluster" resource to list ACLs.
>- Unauthorized requests will receive a ClusterAuthorizationException
>   - This avoids adding a new operation that an existing authorizer
>   implementation may not be aware of.
>   - This can be reviewed and further refined/restricted as a follow
>   up ACLs review after this KIP. See Follow Up Changes
>   
> 
>   .
>5. Requesting a resource or principle that does not have any ACLs will
>not result in an error, instead empty response list is returned
>
> List ACLs Response
>
>
>
> ListAcls Response (Version: 0) => [responses] error_code
>   responses => resource [acls]
> resource => resource_type resource_name
>   resource_type => INT8
>   resource_name => STRING
> acls => acl_principle acl_permission_type acl_host acl_operation
>   acl_principle => STRING
>   acl_permission_type => INT8
>   acl_host => STRING
>   acl_operation => INT8
>   error_code => INT16
>
> Alter ACLs Request
>
>
>
> AlterAcls Request (Version: 0) => [requests]
>   requests => resource [actions]
> resource => resource_type resource_name
>   resource_type => INT8
>   resource_name => STRING
> actions => action acl
>   action => INT8
>   acl => acl_principle acl_permission_type acl_host acl_operation
> acl_principle => STRING
> acl_permission_type => INT8
> acl_host => STRING
> acl_operation => INT8
>
> Request semantics:
>
>1. Must be sent to the controller broker
>2. If there are multiple instructions for the same resource in one
>request an InvalidRequestException will be logged on the broker and a
>single error code for that resource will be returned to the client
>   - This is because the list of requests is modeled server side as a
>   map with resource as the key
>3. ACLs with a delete action will be processed first and the add
>action second.
>1. This is to prevent confusion about sort order and final state when
>   a batch message is sent.
>   2. If an add request was processed first, it could be deleted right
>   after.
>   3. Grouping ACLs by their action allows batching requests to the
>   authorizer via the Authorizer.addAcls and Authorizer.removeAcls calls.
>4. The request is not transactional. One failure wont stop others from
>running.
>   1. If an error occurs on one action, the others could still be run.
>   2. Errors are reported independently.
>   5. The principle must be authorized to the "All" Operation on the
>"Cluster" resource to alter ACLs.
>   - 

[jira] [Commented] (KAFKA-1194) The kafka broker cannot delete the old log files after the configured time

[Ravi Peri (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15377110#comment-15377110
 ] 

Ravi Peri commented on KAFKA-1194:
--

The patch is ready. I am working on few failing unit tests. Once I get a 
100%test pass I will submit a PR.

Get Outlook for Android



On Thu, Jul 14, 2016 at 7:36 AM -0700, "Harald Kirsch (JIRA)" 
> wrote:


[ 
https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fissues.apache.org%2fjira%2fbrowse%2fKAFKA-1194%3fpage%3dcom.atlassian.jira.plugin.system.issuetabpanels%3acomment-tabpanel%26focusedCommentId%3d15377023%23comment-15377023=01%7c01%7craviperi%40microsoft.com%7cce90d4ab0ce5422645ef08d3abf4396e%7c72f988bf86f141af91ab2d7cd011db47%7c1=lbnx5babFf9ysoIpLXxNjHdn8NCoXryf0CcdDVOymtE%3d
 ]

Harald Kirsch commented on KAFKA-1194:
--

Having a very similar problem I would hope that the fix fixes this one too. The 
message and stacktrace is slightly different. We are using the logcleaner with 
compaction and get the below stack trace.

This is on Windows. The claim of the error message that another process has the 
file open is misleading. I verified with procexp and handle search that only 
the Kafka process has the file open, so it is likely blocking itself on this.

Any chance that the patch will fix this one too?
{noformat}
[2016-07-14 16:09:20,568] ERROR [kafka-log-cleaner-thread-0], Error due to  
(kafka.log.LogCleaner)
kafka.common.KafkaStorageException: Failed to change the log file suffix from 
.cleaned to .swap for log segment 0
at kafka.log.LogSegment.kafkaStorageException$1(LogSegment.scala:263)
at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:265)
at kafka.log.Log.replaceSegments(Log.scala:869)
at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:395)
at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:343)
at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:342)
at scala.collection.immutable.List.foreach(List.scala:318)
at kafka.log.Cleaner.clean(LogCleaner.scala:342)
at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:237)
at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:215)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
Caused by: java.nio.file.FileSystemException: 
d:\Search\kafka\__consumer_offsets-40\.log.cleaned -> 
d:\Search\kafka\__consumer_offsets-40\.log.swap: The 
process cannot access the file because it is being used by another process.

at 
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
at 
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:387)
at 
sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
at java.nio.file.Files.move(Files.java:1395)
at 
org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:670)
at kafka.log.FileMessageSet.renameTo(FileMessageSet.scala:364)
... 10 more
Suppressed: java.nio.file.FileSystemException: 
d:\Search\kafka\__consumer_offsets-40\.log.cleaned -> 
d:\Search\kafka\__consumer_offsets-40\.log.swap: The 
process cannot access the file because it is being used by another process.

at 
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
at 
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301)
at 
sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
at java.nio.file.Files.move(Files.java:1395)
at 
org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:667)
... 11 more
{noformat}






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


> The kafka broker cannot delete the old log files after the configured time
> --
>
> Key: KAFKA-1194
> URL: https://issues.apache.org/jira/browse/KAFKA-1194
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.8.1
> Environment: window
>Reporter: Tao Qin
>Assignee: Jay Kreps
>  Labels: features, patch
> Fix For: 0.10.1.0
>
> Attachments: KAFKA-1194.patch, kafka-1194-v1.patch, 
> kafka-1194-v2.patch
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> We tested it in windows environment, and set the log.retention.hours to 24 
> hours.
> 

[jira] [Commented] (KAFKA-1194) The kafka broker cannot delete the old log files after the configured time

[Harald Kirsch (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15377023#comment-15377023
 ] 

Harald Kirsch commented on KAFKA-1194:
--

Having a very similar problem I would hope that the fix fixes this one too. The 
message and stacktrace is slightly different. We are using the logcleaner with 
compaction and get the below stack trace.

This is on Windows. The claim of the error message that another process has the 
file open is misleading. I verified with procexp and handle search that only 
the Kafka process has the file open, so it is likely blocking itself on this.

Any chance that the patch will fix this one too?
{noformat}
[2016-07-14 16:09:20,568] ERROR [kafka-log-cleaner-thread-0], Error due to  
(kafka.log.LogCleaner)
kafka.common.KafkaStorageException: Failed to change the log file suffix from 
.cleaned to .swap for log segment 0
at kafka.log.LogSegment.kafkaStorageException$1(LogSegment.scala:263)
at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:265)
at kafka.log.Log.replaceSegments(Log.scala:869)
at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:395)
at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:343)
at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:342)
at scala.collection.immutable.List.foreach(List.scala:318)
at kafka.log.Cleaner.clean(LogCleaner.scala:342)
at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:237)
at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:215)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
Caused by: java.nio.file.FileSystemException: 
d:\Search\kafka\__consumer_offsets-40\.log.cleaned -> 
d:\Search\kafka\__consumer_offsets-40\.log.swap: The 
process cannot access the file because it is being used by another process.

at 
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
at 
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:387)
at 
sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
at java.nio.file.Files.move(Files.java:1395)
at 
org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:670)
at kafka.log.FileMessageSet.renameTo(FileMessageSet.scala:364)
... 10 more
Suppressed: java.nio.file.FileSystemException: 
d:\Search\kafka\__consumer_offsets-40\.log.cleaned -> 
d:\Search\kafka\__consumer_offsets-40\.log.swap: The 
process cannot access the file because it is being used by another process.

at 
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
at 
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301)
at 
sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
at java.nio.file.Files.move(Files.java:1395)
at 
org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:667)
... 11 more
{noformat}



> The kafka broker cannot delete the old log files after the configured time
> --
>
> Key: KAFKA-1194
> URL: https://issues.apache.org/jira/browse/KAFKA-1194
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.8.1
> Environment: window
>Reporter: Tao Qin
>Assignee: Jay Kreps
>  Labels: features, patch
> Fix For: 0.10.1.0
>
> Attachments: KAFKA-1194.patch, kafka-1194-v1.patch, 
> kafka-1194-v2.patch
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> We tested it in windows environment, and set the log.retention.hours to 24 
> hours.
> # The minimum age of a log file to be eligible for deletion
> log.retention.hours=24
> After several days, the kafka broker still cannot delete the old log file. 
> And we get the following exceptions:
> [2013-12-19 01:57:38,528] ERROR Uncaught exception in scheduled task 
> 'kafka-log-retention' (kafka.utils.KafkaScheduler)
> kafka.common.KafkaStorageException: Failed to change the log file suffix from 
>  to .deleted for log segment 1516723
>  at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:249)
>  at kafka.log.Log.kafka$log$Log$$asyncDeleteSegment(Log.scala:638)
>  at kafka.log.Log.kafka$log$Log$$deleteSegment(Log.scala:629)
>  at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:418)
>  at 

[jira] [Commented] (KAFKA-3894) Log Cleaner thread crashes and never restarts

[Tom Crayford (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15376858#comment-15376858
 ] 

Tom Crayford commented on KAFKA-3894:
-

Jun:

#4 seems potentially very complex to me. It also doesn't work in the case that 
the broker is shut down and the dedupe buffer size adjusted. I much prefer #3 - 
it maps fine into the existing model as far as I can tell - we'd "just" split 
the log file we're cleaning once the offsetmap is full. That of course requires 
a little more IO, but it doesn't involve implementing (or using a library for) 
sketches that could potentially be incorrect. It also seems like the right long 
term solution, and more robust than automatically rolling log files some of the 
time. Am I missing something here? 

Upsides of #3 vs #4:
We can now clean the largest log segment, no matter the buffer size.
We don't increase complexity of the produce path, or change memory usage.
We don't have to implement or reuse a library for estimating unique keys
We don't have to figure out storing the key estimate (e.g. in the index or in a 
new file alongside each segment).

Downsides:
It would increase the complexity of the cleaner.
The code that swaps in and out segments will also get more complex, and the 
crash-safety of that code is already tricky.

Exists in both:
Larger log segments could potentially be split a lot, and not always 
deduplicated that well together. For example, if I write the max number of 
unique keys for the offset map into a topic, then the segment rolls, then I 
write a tombstone for every message in the previously sent messages, then 
neither #3 nor #4 would ever clear up any data. This is no worse than today 
though.

Cassandra and other LSM based systems that do log structured storage and 
over-time compaction use similar "splitting and combining" mechanisms to ensure 
everything gets cleared up over time without using too much memory. They have a 
very different storage architecture and goals to Kafka's compaction, for sure, 
but it's interesting to note that they care about similar things.

> Log Cleaner thread crashes and never restarts
> -
>
> Key: KAFKA-3894
> URL: https://issues.apache.org/jira/browse/KAFKA-3894
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2.2, 0.9.0.1
> Environment: Oracle JDK 8
> Ubuntu Precise
>Reporter: Tim Carey-Smith
>  Labels: compaction
>
> The log-cleaner thread can crash if the number of keys in a topic grows to be 
> too large to fit into the dedupe buffer. 
> The result of this is a log line: 
> {quote}
> broker=0 pri=ERROR t=kafka-log-cleaner-thread-0 at=LogCleaner 
> \[kafka-log-cleaner-thread-0\], Error due to  
> java.lang.IllegalArgumentException: requirement failed: 9750860 messages in 
> segment MY_FAVORITE_TOPIC-2/47580165.log but offset map can fit 
> only 5033164. You can increase log.cleaner.dedupe.buffer.size or decrease 
> log.cleaner.threads
> {quote}
> As a result, the broker is left in a potentially dangerous situation where 
> cleaning of compacted topics is not running. 
> It is unclear if the broader strategy for the {{LogCleaner}} is the reason 
> for this upper bound, or if this is a value which must be tuned for each 
> specific use-case. 
> Of more immediate concern is the fact that the thread crash is not visible 
> via JMX or exposed as some form of service degradation. 
> Some short-term remediations we have made are:
> * increasing the size of the dedupe buffer
> * monitoring the log-cleaner threads inside the JVM



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk8 #752

[Apache Jenkins Server]

See 

Changes:

[ismael] KAFKA-3905: Handling null/empty topics and collections, patterns when

--
[...truncated 4744 lines...]

kafka.utils.ZkUtilsTest > testSuccessfulConditionalDeletePath PASSED

kafka.utils.ByteBoundedBlockingQueueTest > testByteBoundedBlockingQueue STARTED

kafka.utils.ByteBoundedBlockingQueueTest > testByteBoundedBlockingQueue PASSED

kafka.utils.ReplicationUtilsTest > testUpdateLeaderAndIsr STARTED

kafka.utils.ReplicationUtilsTest > testUpdateLeaderAndIsr PASSED

kafka.utils.ReplicationUtilsTest > testGetLeaderIsrAndEpochForPartition STARTED

kafka.utils.ReplicationUtilsTest > testGetLeaderIsrAndEpochForPartition PASSED

kafka.utils.timer.TimerTest > testAlreadyExpiredTask STARTED

kafka.utils.timer.TimerTest > testAlreadyExpiredTask PASSED

kafka.utils.timer.TimerTest > testTaskExpiration STARTED

kafka.utils.timer.TimerTest > testTaskExpiration PASSED

kafka.utils.timer.TimerTaskListTest > testAll STARTED

kafka.utils.timer.TimerTaskListTest > testAll PASSED

kafka.utils.SchedulerTest > testMockSchedulerNonPeriodicTask STARTED

kafka.utils.SchedulerTest > testMockSchedulerNonPeriodicTask PASSED

kafka.utils.SchedulerTest > testMockSchedulerPeriodicTask STARTED

kafka.utils.SchedulerTest > testMockSchedulerPeriodicTask PASSED

kafka.utils.SchedulerTest > testNonPeriodicTask STARTED

kafka.utils.SchedulerTest > testNonPeriodicTask PASSED

kafka.utils.SchedulerTest > testRestart STARTED

kafka.utils.SchedulerTest > testRestart PASSED

kafka.utils.SchedulerTest > testReentrantTaskInMockScheduler STARTED

kafka.utils.SchedulerTest > testReentrantTaskInMockScheduler PASSED

kafka.utils.SchedulerTest > testPeriodicTask STARTED

kafka.utils.SchedulerTest > testPeriodicTask PASSED

kafka.utils.CommandLineUtilsTest > testParseEmptyArg STARTED

kafka.utils.CommandLineUtilsTest > testParseEmptyArg PASSED

kafka.utils.CommandLineUtilsTest > testParseSingleArg STARTED

kafka.utils.CommandLineUtilsTest > testParseSingleArg PASSED

kafka.utils.CommandLineUtilsTest > testParseArgs STARTED

kafka.utils.CommandLineUtilsTest > testParseArgs PASSED

kafka.utils.CommandLineUtilsTest > testParseEmptyArgAsValid STARTED

kafka.utils.CommandLineUtilsTest > testParseEmptyArgAsValid PASSED

kafka.utils.UtilsTest > testAbs STARTED

kafka.utils.UtilsTest > testAbs PASSED

kafka.utils.UtilsTest > testReplaceSuffix STARTED

kafka.utils.UtilsTest > testReplaceSuffix PASSED

kafka.utils.UtilsTest > testCircularIterator STARTED

kafka.utils.UtilsTest > testCircularIterator PASSED

kafka.utils.UtilsTest > testReadBytes STARTED

kafka.utils.UtilsTest > testReadBytes PASSED

kafka.utils.UtilsTest > testCsvList STARTED

kafka.utils.UtilsTest > testCsvList PASSED

kafka.utils.UtilsTest > testReadInt STARTED

kafka.utils.UtilsTest > testReadInt PASSED

kafka.utils.UtilsTest > testCsvMap STARTED

kafka.utils.UtilsTest > testCsvMap PASSED

kafka.utils.UtilsTest > testInLock STARTED

kafka.utils.UtilsTest > testInLock PASSED

kafka.utils.UtilsTest > testSwallow STARTED

kafka.utils.UtilsTest > testSwallow PASSED

kafka.utils.JsonTest > testJsonEncoding STARTED

kafka.utils.JsonTest > testJsonEncoding PASSED

kafka.utils.IteratorTemplateTest > testIterator STARTED

kafka.utils.IteratorTemplateTest > testIterator PASSED

kafka.metrics.KafkaTimerTest > testKafkaTimer STARTED

kafka.metrics.KafkaTimerTest > testKafkaTimer PASSED

kafka.metrics.MetricsTest > testMetricsReporterAfterDeletingTopic STARTED

kafka.metrics.MetricsTest > testMetricsReporterAfterDeletingTopic PASSED

kafka.metrics.MetricsTest > 
testBrokerTopicMetricsUnregisteredAfterDeletingTopic STARTED

kafka.metrics.MetricsTest > 
testBrokerTopicMetricsUnregisteredAfterDeletingTopic PASSED

kafka.metrics.MetricsTest > testMetricsLeak STARTED

kafka.metrics.MetricsTest > testMetricsLeak PASSED

kafka.consumer.TopicFilterTest > testWhitelists STARTED

kafka.consumer.TopicFilterTest > testWhitelists PASSED

kafka.consumer.TopicFilterTest > 
testWildcardTopicCountGetTopicCountMapEscapeJson STARTED

kafka.consumer.TopicFilterTest > 
testWildcardTopicCountGetTopicCountMapEscapeJson PASSED

kafka.consumer.TopicFilterTest > testBlacklists STARTED

kafka.consumer.TopicFilterTest > testBlacklists PASSED

kafka.consumer.PartitionAssignorTest > testRoundRobinPartitionAssignor STARTED

kafka.consumer.PartitionAssignorTest > testRoundRobinPartitionAssignor PASSED

kafka.consumer.PartitionAssignorTest > testRangePartitionAssignor STARTED

kafka.consumer.PartitionAssignorTest > testRangePartitionAssignor PASSED

kafka.consumer.ZookeeperConsumerConnectorTest > testBasic STARTED

kafka.consumer.ZookeeperConsumerConnectorTest > testBasic PASSED

kafka.consumer.ZookeeperConsumerConnectorTest > testCompressionSetConsumption 
STARTED

kafka.consumer.ZookeeperConsumerConnectorTest > testCompressionSetConsumption 
PASSED

kafka.consumer.ZookeeperConsumerConnectorTest > 

[jira] [Resolved] (KAFKA-3905) Check for null in KafkaConsumer#{subscribe, assign}

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-3905.

   Resolution: Fixed
 Reviewer: Ismael Juma
Fix Version/s: 0.10.1.0

> Check for null in KafkaConsumer#{subscribe, assign}
> ---
>
> Key: KAFKA-3905
> URL: https://issues.apache.org/jira/browse/KAFKA-3905
> Project: Kafka
>  Issue Type: Wish
>  Components: clients
>Affects Versions: 0.10.0.0
>Reporter: Xing Huang
>Assignee: Rekha Joshi
>Priority: Minor
> Fix For: 0.10.1.0
>
>
> Currently, KafkaConsumer's subscribe methods accept Collection as 
> topics to be subscribed, but a Collection may have null as its element. For 
> example
> {code}
> String topic = null;
> Collection topics = Arrays.asList(topic);
> consumer.subscribe(topics)
> {code}
> When this happens, consumer will throw a puzzling NullPointerException:
> {code}
>   at org.apache.kafka.common.utils.Utils.utf8Length(Utils.java:245)
>   at org.apache.kafka.common.protocol.types.Type$6.sizeOf(Type.java:248)
>   at 
> org.apache.kafka.common.protocol.types.ArrayOf.sizeOf(ArrayOf.java:85)
>   at org.apache.kafka.common.protocol.types.Schema.sizeOf(Schema.java:89)
>   at org.apache.kafka.common.protocol.types.Struct.sizeOf(Struct.java:244)
>   at 
> org.apache.kafka.common.requests.RequestSend.serialize(RequestSend.java:35)
>   at 
> org.apache.kafka.common.requests.RequestSend.(RequestSend.java:29)
>   at 
> org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.request(NetworkClient.java:616)
>   at 
> org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:639)
>   at 
> org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:552)
>   at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:258)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:360)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:224)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:192)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:163)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:179)
>   at 
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:970)
>   at 
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:934)
> {code}
> Maybe it's better to remove null when doing subscription.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3905) Check for null in KafkaConsumer#{subscribe, assign}

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3905:
---
Summary: Check for null in KafkaConsumer#{subscribe, assign}  (was: remove 
null from subscribed topics  in KafkaConsumer#subscribe)

> Check for null in KafkaConsumer#{subscribe, assign}
> ---
>
> Key: KAFKA-3905
> URL: https://issues.apache.org/jira/browse/KAFKA-3905
> Project: Kafka
>  Issue Type: Wish
>  Components: clients
>Affects Versions: 0.10.0.0
>Reporter: Xing Huang
>Assignee: Rekha Joshi
>Priority: Minor
> Fix For: 0.10.1.0
>
>
> Currently, KafkaConsumer's subscribe methods accept Collection as 
> topics to be subscribed, but a Collection may have null as its element. For 
> example
> {code}
> String topic = null;
> Collection topics = Arrays.asList(topic);
> consumer.subscribe(topics)
> {code}
> When this happens, consumer will throw a puzzling NullPointerException:
> {code}
>   at org.apache.kafka.common.utils.Utils.utf8Length(Utils.java:245)
>   at org.apache.kafka.common.protocol.types.Type$6.sizeOf(Type.java:248)
>   at 
> org.apache.kafka.common.protocol.types.ArrayOf.sizeOf(ArrayOf.java:85)
>   at org.apache.kafka.common.protocol.types.Schema.sizeOf(Schema.java:89)
>   at org.apache.kafka.common.protocol.types.Struct.sizeOf(Struct.java:244)
>   at 
> org.apache.kafka.common.requests.RequestSend.serialize(RequestSend.java:35)
>   at 
> org.apache.kafka.common.requests.RequestSend.(RequestSend.java:29)
>   at 
> org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.request(NetworkClient.java:616)
>   at 
> org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:639)
>   at 
> org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:552)
>   at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:258)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:360)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:224)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:192)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:163)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:179)
>   at 
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:970)
>   at 
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:934)
> {code}
> Maybe it's better to remove null when doing subscription.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3905) remove null from subscribed topics in KafkaConsumer#subscribe

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15376692#comment-15376692
 ] 

ASF GitHub Bot commented on KAFKA-3905:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1601


> remove null from subscribed topics  in KafkaConsumer#subscribe
> --
>
> Key: KAFKA-3905
> URL: https://issues.apache.org/jira/browse/KAFKA-3905
> Project: Kafka
>  Issue Type: Wish
>  Components: clients
>Affects Versions: 0.10.0.0
>Reporter: Xing Huang
>Assignee: Rekha Joshi
>Priority: Minor
>
> Currently, KafkaConsumer's subscribe methods accept Collection as 
> topics to be subscribed, but a Collection may have null as its element. For 
> example
> {code}
> String topic = null;
> Collection topics = Arrays.asList(topic);
> consumer.subscribe(topics)
> {code}
> When this happens, consumer will throw a puzzling NullPointerException:
> {code}
>   at org.apache.kafka.common.utils.Utils.utf8Length(Utils.java:245)
>   at org.apache.kafka.common.protocol.types.Type$6.sizeOf(Type.java:248)
>   at 
> org.apache.kafka.common.protocol.types.ArrayOf.sizeOf(ArrayOf.java:85)
>   at org.apache.kafka.common.protocol.types.Schema.sizeOf(Schema.java:89)
>   at org.apache.kafka.common.protocol.types.Struct.sizeOf(Struct.java:244)
>   at 
> org.apache.kafka.common.requests.RequestSend.serialize(RequestSend.java:35)
>   at 
> org.apache.kafka.common.requests.RequestSend.(RequestSend.java:29)
>   at 
> org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.request(NetworkClient.java:616)
>   at 
> org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:639)
>   at 
> org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:552)
>   at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:258)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:360)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:224)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:192)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:163)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:179)
>   at 
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:970)
>   at 
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:934)
> {code}
> Maybe it's better to remove null when doing subscription.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1601: KAFKA-3905; Handle invalid collection of topics, p...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1601


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (KAFKA-3965) Mirror maker sync send fails will lose messages

[NieWang (JIRA)]

NieWang created KAFKA-3965:
--

 Summary: Mirror maker sync send fails will lose messages
 Key: KAFKA-3965
 URL: https://issues.apache.org/jira/browse/KAFKA-3965
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.10.0.0
 Environment: SUSE Linux Enterprise Server 11 (x86_64)
Reporter: NieWang
 Fix For: 0.10.1.0


1、Source kafka cluster storage some messages and message size is 500 bytes.
2、Mirror maker producer config 
 producer.type=sync
 max.request.size=400
3、Start mirror maker backup message from source kafka cluster to destination 
kafka cluster, then mirror maker will quit because message larger than 400.
4、Check source kafka cluster will find offset have set to 1.
5、Check destination kafka cluster, then find have not any message.
6、Delete producer config max.request.size=400，start mirror maker again. Wait 
mirror maker finish backup, then find have lose the first message.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Kafka Streams question

[Michael Noll]

Also, in the next version of Kafka / Kafka Streams such "intermediate"
topics will automatically be created for you when you do joins or
aggregations:

https://issues.apache.org/jira/browse/KAFKA-3561

So my previous message explained your options today when using the current
release of Kafka Streams (v0.10.0.0).

-Michael




On Thu, Jul 14, 2016 at 10:32 AM, Michael Noll  wrote:

> Poul,
>
> to add to what Matthias said:  If you are wondering how to manually create
> a topic, you have basically two options.
>
> A. Use Kafka's CLI tools to create the topic "from the outside".
>
> # Example
> $ kafka-topics.sh --create --topic my-custom-toipc --zookeeper
> localhost:2181 --partitions 1 --replication-factor 1
>
> B. Use Kafka's API to programmatically create the topic.  See [1] for an
> example.
>
> Question for you to learn how we could perhaps improve the status quo:
>  How would you have expected this to work in the current Kafka Streams
> API?  For example, would you have expected that, say, the `through()`
> method would accept parameters to specify the number of partitions?
>
>
> Hope this helps,
> Michael
>
> [1]
> https://github.com/confluentinc/examples/blob/kafka-0.10.0.0-cp-3.0.0/kafka-streams/src/test/java/io/confluent/examples/streams/kafka/KafkaEmbedded.java#L133-L160
>
>
>
>
> On Thu, Jul 14, 2016 at 10:08 AM, Matthias J. Sax 
> wrote:
>
>> Hi,
>>
>> you can manually create a topic with the number of partitions you want
>> to have and use this topic via through()
>>
>> KStream input = ...
>>
>> input.map().through("manually-created-topic").join(...)
>>
>> However, both KStream and KTable need to have the same number of
>> partitions for perform the join. Thus, you might need to create a topic
>> (with the same number of partitions) for the table, too.
>>
>> See
>>
>> http://docs.confluent.io/3.0.0/streams/developer-guide.html#joining-streams
>>
>>
>> -Matthias
>>
>> On 07/13/2016 11:59 PM, Poul Costinsky wrote:
>> > Hi! I am prototyping some code using Kafka Streams, and have a
>> question. I need to map a stream into another (with different partition
>> key) and join it with a table. How do I control number of partitions of the
>> mapped stream?
>> >
>> > Thanks!
>> >
>> > Poul Costinsky
>> > Chief Architect
>> >
>> >  
>> > (360) 207-1753 
>> >
>> >
>> >
>> >
>> >
>>
>>
>
>
> --
>
> *Michael G. Noll | Product Manager | Confluent | +1 650.453.5860
> <%2B1%20650.453.5860>Download Apache Kafka and Confluent Platform:
> www.confluent.io/download *
>



-- 

*Michael G. Noll | Product Manager | Confluent | +1 650.453.5860Download
Apache Kafka and Confluent Platform: www.confluent.io/download
*

[GitHub] kafka pull request #1623: MINOR: Fix zk inconsistent security settings check

[ijuma]

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/1623

MINOR: Fix zk inconsistent security settings check

Also include a few minor clean-ups.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka fix-zk-inconsistent-security-check

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1623.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1623


commit ee43957fb82035e899d86b4768316210d071fdd5
Author: Ismael Juma 
Date:   2016-07-14T08:33:33Z

Fix check for inconsistent ZK settings

The previous check always evaluated to `true`.

commit 370889d91db66c7f4ef29e268e4e4542ea7b4a3b
Author: Ismael Juma 
Date:   2016-07-14T08:37:24Z

A few minor clean-ups in `KafkaServer.initZk()`




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Kafka Streams question

[Michael Noll]

Poul,

to add to what Matthias said:  If you are wondering how to manually create
a topic, you have basically two options.

A. Use Kafka's CLI tools to create the topic "from the outside".

# Example
$ kafka-topics.sh --create --topic my-custom-toipc --zookeeper
localhost:2181 --partitions 1 --replication-factor 1

B. Use Kafka's API to programmatically create the topic.  See [1] for an
example.

Question for you to learn how we could perhaps improve the status quo:  How
would you have expected this to work in the current Kafka Streams API?  For
example, would you have expected that, say, the `through()` method would
accept parameters to specify the number of partitions?


Hope this helps,
Michael

[1]
https://github.com/confluentinc/examples/blob/kafka-0.10.0.0-cp-3.0.0/kafka-streams/src/test/java/io/confluent/examples/streams/kafka/KafkaEmbedded.java#L133-L160




On Thu, Jul 14, 2016 at 10:08 AM, Matthias J. Sax 
wrote:

> Hi,
>
> you can manually create a topic with the number of partitions you want
> to have and use this topic via through()
>
> KStream input = ...
>
> input.map().through("manually-created-topic").join(...)
>
> However, both KStream and KTable need to have the same number of
> partitions for perform the join. Thus, you might need to create a topic
> (with the same number of partitions) for the table, too.
>
> See
> http://docs.confluent.io/3.0.0/streams/developer-guide.html#joining-streams
>
>
> -Matthias
>
> On 07/13/2016 11:59 PM, Poul Costinsky wrote:
> > Hi! I am prototyping some code using Kafka Streams, and have a question.
> I need to map a stream into another (with different partition key) and join
> it with a table. How do I control number of partitions of the mapped stream?
> >
> > Thanks!
> >
> > Poul Costinsky
> > Chief Architect
> >
> >  
> > (360) 207-1753 
> >
> >
> >
> >
> >
>
>


-- 

*Michael G. Noll | Product Manager | Confluent | +1 650.453.5860Download
Apache Kafka and Confluent Platform: www.confluent.io/download
*

Re: Kafka Streams question

[Matthias J. Sax]

Hi,

you can manually create a topic with the number of partitions you want
to have and use this topic via through()

KStream input = ...

input.map().through("manually-created-topic").join(...)

However, both KStream and KTable need to have the same number of
partitions for perform the join. Thus, you might need to create a topic
(with the same number of partitions) for the table, too.

See
http://docs.confluent.io/3.0.0/streams/developer-guide.html#joining-streams


-Matthias

On 07/13/2016 11:59 PM, Poul Costinsky wrote:
> Hi! I am prototyping some code using Kafka Streams, and have a question. I 
> need to map a stream into another (with different partition key) and join it 
> with a table. How do I control number of partitions of the mapped stream?
> 
> Thanks! 
> 
> Poul Costinsky
> Chief Architect
> 
>  
> (360) 207-1753 
> 
> 
> 
> 
> 



signature.asc
Description: OpenPGP digital signature