[jira] [Updated] (KAFKA-3147) Memory records is not writable in MirrorMaker

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3147:
---
Status: Patch Available  (was: Open)

> Memory records is not writable in MirrorMaker
> -
>
> Key: KAFKA-3147
> URL: https://issues.apache.org/jira/browse/KAFKA-3147
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.0
>Reporter: Meghana Narasimhan
>Assignee: Mayuresh Gharat
>
> Hi,
> We are running a 3 node cluster (kafka version 0.9) and Node 0 also has a few 
> mirror makers running. 
> When we do a rolling restart of the cluster, the mirror maker shuts down with 
> the following errors.
> [2016-01-11 20:16:00,348] WARN Got error produce response with correlation id 
> 12491674 on topic-partition test-99, retrying (2147483646 attempts left). 
> Error: NOT_LEADER_FOR_PARTITION 
> (org.apache.kafka.clients.producer.internals.Sender)
> [2016-01-11 20:16:00,853] FATAL [mirrormaker-thread-0] Mirror maker thread 
> failure due to  (kafka.tools.MirrorMaker$MirrorMakerThread)
> java.lang.IllegalStateException: Memory records is not writable
> at 
> org.apache.kafka.common.record.MemoryRecords.append(MemoryRecords.java:93)
> at 
> org.apache.kafka.clients.producer.internals.RecordBatch.tryAppend(RecordBatch.java:69)
> at 
> org.apache.kafka.clients.producer.internals.RecordAccumulator.append(RecordAccumulator.java:168)
> at 
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:435)
> at 
> kafka.tools.MirrorMaker$MirrorMakerProducer.send(MirrorMaker.scala:593)
> at 
> kafka.tools.MirrorMaker$MirrorMakerThread$$anonfun$run$3.apply(MirrorMaker.scala:398)
> at 
> kafka.tools.MirrorMaker$MirrorMakerThread$$anonfun$run$3.apply(MirrorMaker.scala:398)
> at scala.collection.Iterator$class.foreach(Iterator.scala:742)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at 
> kafka.tools.MirrorMaker$MirrorMakerThread.run(MirrorMaker.scala:398)
> [2016-01-11 20:16:01,072] WARN Got error produce response with correlation id 
> 12491679 on topic-partition test-75, retrying (2147483646 attempts left). 
> Error: NETWORK_EXCEPTION (org.apache.kafka.clients.producer.internals.Sender)
> [2016-01-11 20:16:01,073] WARN Got error produce response with correlation id 
> 12491679 on topic-partition test-93, retrying (2147483646 attempts left). 
> Error: NETWORK_EXCEPTION (org.apache.kafka.clients.producer.internals.Sender)
> [2016-01-11 20:16:01,073] WARN Got error produce response with correlation id 
> 12491679 on topic-partition test-24, retrying (2147483646 attempts left). 
> Error: NETWORK_EXCEPTION (org.apache.kafka.clients.producer.internals.Sender)
> [2016-01-11 20:16:20,479] FATAL [mirrormaker-thread-0] Mirror maker thread 
> exited abnormally, stopping the whole mirror maker. 
> (kafka.tools.MirrorMaker$MirrorMakerThread)
> Curious if the NOT_LEADER_FOR_PARTITION is because of a potential bug hinted 
> at in the thread , 
> http://mail-archives.apache.org/mod_mbox/kafka-users/201505.mbox/%3ccajs3ho_u8s1xou_kudnfjamypjtmrjlw10qvkngn2yqkdan...@mail.gmail.com%3E
>
> And I think the mirror maker shuts down because of the 
> "abort.on.send.failure" which is set to true in our case. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-43: Kafka SASL enhancements

[tao xiao]

Sorry, just ignore previous email. I saw the newly defined interface of the
callback in the KIP which has considered this matter.

On Fri, 29 Jan 2016 at 18:08 tao xiao  wrote:

> Hi Rajini,
>
> Do you consider exposing Subject to AuthCallback as well? It is useful for
> users building their own SASL mechanism so that we have control  where to
> put logon data in subject and how to manipulate in SASL callback
>
>
> On Fri, 29 Jan 2016 at 18:04 Rajini Sivaram 
> wrote:
>
>> Following on from the KIP meeting on Tuesday, I have updated the KIP with
>> a
>> flow for negotiation of mechanisms to support multiple SASL mechanisms
>> within a broker. I have also added a configurable Login interface to
>> support custom mechanisms which require ticket refresh - requested by Tao
>> Xiao.
>>
>> I will work on updating the PR in KAFKA-3149 over the next few days since
>> it will be useful for review.
>>
>> All comments and suggestions are welcome.
>>
>>
>> On Thu, Jan 28, 2016 at 2:35 PM, tao xiao  wrote:
>>
>> > Sounds like a good approach to add provider in login module. Would love
>> to
>> > see updates in the PR to reflect the changes in Login and
>> > AuthCallbackHandler.
>> >
>> > On Thu, 28 Jan 2016 at 19:31 Rajini Sivaram <
>> rajinisiva...@googlemail.com>
>> > wrote:
>> >
>> > > Tao,
>> > >
>> > > We currently add the security provider in a static initializer in our
>> > login
>> > > module. This ensures that the security provider is always installed
>> > before
>> > > Kafka creates SaslServer/SaslClient. As you say, it is also possible
>> to
>> > > insert code into your application to add security provider before
>> Kafka
>> > > clients are created. Since you can also configure the JDK to add new
>> > > security providers, I am not sure if there is value in adding more
>> > > configuration in Kafka to add security providers.
>> > >
>> > > On Thu, Jan 28, 2016 at 2:25 AM, tao xiao 
>> wrote:
>> > >
>> > > > The callback works for me as long as it has access to Subject and
>> > mechs.
>> > > > The other thing is how we can inject the customized security
>> provider
>> > via
>> > > > Security.addProvider()? If I want to implement my own SASL mech I
>> need
>> > to
>> > > > call the addProvider() before SASL.create so that my own
>> implementation
>> > > of
>> > > > SASLClient/Sever can be returned. Any thoughts on this? we can
>> either
>> > let
>> > > > users inject the provider in their logic code before creating a
>> > > > producer/consumer or Kafka does it for users
>> > > >
>> > > > On Thu, 28 Jan 2016 at 03:36 Rajini Sivaram <
>> > > rajinisiva...@googlemail.com>
>> > > > wrote:
>> > > >
>> > > > > Hi Tao,
>> > > > >
>> > > > > *javax.security.auth.callback.**CallbackHandler *is the standard
>> way
>> > in
>> > > > > which SASL clients and server obtain additional mechanism specific
>> > > > > input. *AuthCallbackHandler
>> > > > > *simply extends this interface to propagate configuration
>> > properties. I
>> > > > was
>> > > > > going to provide SASL mechanism and Subject to the callback
>> handlers
>> > as
>> > > > > well since the default handlers use these.
>> > > > >
>> > > > > Your SaslServer/SaslClient implementation can obtain the Subject
>> > using
>> > > > > *Subject.getSubject(**AccessController.getContext(). *But it will
>> be
>> > > good
>> > > > > to know if callback handlers would work for you - apart from
>> standard
>> > > > > callbacks like PasswordCallback, you can define your own callbacks
>> > too
>> > > if
>> > > > > you require.
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Wed, Jan 27, 2016 at 3:59 PM, tao xiao 
>> > > wrote:
>> > > > >
>> > > > > > Thanks Rajini. The other thing in my mind is that we should
>> find a
>> > > way
>> > > > to
>> > > > > > expose subject to SASL so that other mechanisms are able to use
>> the
>> > > > > > principal and credentials stored in subject to do
>> authentication.
>> > > > > >
>> > > > > > I am thinking to have below interface that can be extended by
>> users
>> > > to
>> > > > > > build the SASL client/server instead of having an AuthCallback.
>> > With
>> > > > this
>> > > > > > interface users are able to add their own security provider
>> before
>> > > > > > client/server is returned by SASL. Any thoughts?
>> > > > > >
>> > > > > > Interface SaslClientBuilder {
>> > > > > >
>> > > > > > SaslClient build(mechs, subject, host, otherparams)
>> > > > > > }
>> > > > > >
>> > > > > > Interface SaslServerBuilder {
>> > > > > > SaslServer build(mechs, subject, host, otherparams)
>> > > > > > }
>> > > > > >
>> > > > > > On Wed, 27 Jan 2016 at 18:54 Rajini Sivaram <
>> > > > > rajinisiva...@googlemail.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Tao,
>> > > > > > >
>> > > > > > > Thank you for the explanation. I couldn't find a standard Java
>> > > > > interface
>> > > > > > > that would be suitable, so will 

[GitHub] kafka pull request: KAFKA-3170: Set default fetch_min_bytes in new...

[rajinisivaram]

GitHub user rajinisivaram opened a pull request:

https://github.com/apache/kafka/pull/832

KAFKA-3170: Set default fetch_min_bytes in new consumer to 1

Set default to 1 instead of 1024, this matches the existing doc and feels 
like a better default value. Have run the unit tests with the change.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rajinisivaram/kafka KAFKA-3170

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/832.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #832


commit 7e9f944a61e331738ff7c9ed7ee93495039d697f
Author: Rajini Sivaram 
Date:   2016-01-29T13:47:08Z

KAFKA-3170: Set default fetch_min_bytes in new consumer to 1




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Does kstreams have a key-value store that is backed by Kafka?

[Neha Narkhede]

James,

That's correct. You can find the current Kafka Streams code under the
streams package in Kafka trunk. The state store API is at
org.apache.kafka.streams.state. There are examples of using the low-level
processor API as well as the DSL under org.apache.kafka.streams.examples
and more are coming soon.

Thanks,
Neha

On Thu, Jan 28, 2016 at 11:01 PM, James Cheng  wrote:

> Hi,
>
> I think I've heard that the upcoming Kafka Streams has something like
> Samza's local state key-value database. That is, something that looks like
> a key-value store that saves all writes to Kafka, and that can then later
> be reinitialized from Kafka.
>
> Does such a thing exist? Can someone point me to it? I'd like to learn
> more about it.
>
> Thanks,
> -James
>
>
> 
>
> This email and any attachments may contain confidential and privileged
> material for the sole use of the intended recipient. Any review, copying,
> or distribution of this email (or any attachments) by others is prohibited.
> If you are not the intended recipient, please contact the sender
> immediately and permanently delete this email and any attachments. No
> employee or agent of TiVo Inc. is authorized to conclude any binding
> agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
> Inc. may only be made by a signed written agreement.
>



-- 
Thanks,
Neha

Re: [DISCUSS] KIP-42: Add Producer and Consumer Interceptors

[Neha Narkhede]

Becket,

Is your concern the presence of CRC in the RecordMetadata or do you want to
brainstorm how CRC can be used for auditing? I think we shouldn't try to
think about the various ways that people can do monitoring using
interceptors and the metadata we provide. The entire point of having
pluggable interceptors is so that people can employ their own creative
mechanisms to make use of interceptors.

I do think that it is worth discussing whether or not CRC makes sense as
record metadata to the user. My take is that the CRC is the best size-bound
summary of serialized record content available to us which is expensive to
recompute if the user were to redo it. I'd argue this summary of a record
qualifies as its metadata. After all, we use the record CRC for a very
important test of the system durability as it travels through the system.

1. Isn't the TopicPartition + Offset already uniquely identified a message?
> It seems better than CRC no matter from summary point of view or auditing
> point of view.


The offset is a system-assigned value of uniqueness to the message. If you
trusted the system that much, you are not looking to monitor it out-of-band
:-)


> 2. Currently CRC only has 4 bytes. So it will have collision when there are
> more than ~4 billion messages. Take LinkedIn as an example, we have 1.3
> trillion messages per day. So there will be at least a couple of hundreds
> collision for each CRC code every day, whereas TopicPartition+Offset will
> not have any collision.


The CRC isn't sent over the wire and doesn't add any extra overhead in
processing, so what is your concern? If you aren't convinced about its
usefulness, you can always use the default do-nothing interceptor at
LinkedIn and ignore the CRC.

Without having
> the entire message bytes, they may not be able to verify its correctness,
> and the CRC could even be invalid if the broker ever overwritten any field
> or did format conversion.
>

This doesn't make sense to me. The CRC is used for the most important
durability check by Kafka - to verify that the message was not garbled over
the wire. The system can't change it; it has to match on the consumer side
or we will not return it to the user.

On Fri, Jan 29, 2016 at 3:23 AM, Becket Qin  wrote:

> Anna,
>
> It is still not clear to me why we should expose CRC to end user.
> Followings are my confusions.
>
> 1. Isn't the TopicPartition + Offset already uniquely identified a message?
> It seems better than CRC no matter from summary point of view or auditing
> point of view.
>
> 2. Currently CRC only has 4 bytes. So it will have collision when there are
> more than ~4 billion messages. Take LinkedIn as an example, we have 1.3
> trillion messages per day. So there will be at least a couple of hundreds
> collision for each CRC code every day, whereas TopicPartition+Offset will
> not have any collision.
>
> 3. CRC is calculated after all the fields have been filled in by producer,
> including timestamp, attributes, etc. It might also get recomputed on
> broker side. So if users only get CRC from record metadata. Without having
> the entire message bytes, they may not be able to verify its correctness,
> and the CRC could even be invalid if the broker ever overwritten any field
> or did format conversion.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
>
>
>
> On Thu, Jan 28, 2016 at 5:58 PM, Anna Povzner  wrote:
>
> > On a second thought, yes, I think we should expose record size that
> > represents application bytes. This is Becket's option #1.
> >
> > I updated the KIP wiki with new fields in RecordMetadata and
> > ConsumerRecord.
> >
> > I would like to start a voting thread tomorrow if there are no objections
> > or more things to discuss.
> >
> > Thanks,
> > Anna
> >
> >
> >
> > On Thu, Jan 28, 2016 at 2:25 PM, Anna Povzner  wrote:
> >
> > > Regarding record size as bytes sent over the wire, my concern is that
> it
> > > is almost impossible to calculate per-record. We could do as: 1)
> > compressed
> > > bytes / number of records in a compressed message, as Todd mentioned;
> or
> > 2)
> > > or same as #1 but take it proportional to uncompressed record size vs.
> > > total uncompressed size of records. All of these calculations give us
> an
> > > estimate. So maybe record size as bytes sent over the wire is not a
> > > per-record metadata, but rather per topic/partition measure that is
> > better
> > > to be exposed through metrics?
> > >
> > >
> > > On Thu, Jan 28, 2016 at 2:09 PM, Todd Palino 
> wrote:
> > >
> > >> It may be difficult (or nearly impossible) to get actual compressed
> > bytes
> > >> for a message from a compressed batch, but I do think it’s useful
> > >> information to have available for the very reason noted, bandwidth
> > >> consumed. Does it make sense to have an interceptor at the batch level
> > >> that
> > >> can provide this? The other option is to estimate it (such as making
> an
> > 

[jira] [Created] (KAFKA-3170) Default value of fetch_min_bytes in new consumer is 1024 while doc says it is 1

[Rajini Sivaram (JIRA)]

Rajini Sivaram created KAFKA-3170:
-

 Summary: Default value of fetch_min_bytes in new consumer is 1024 
while doc says it is 1
 Key: KAFKA-3170
 URL: https://issues.apache.org/jira/browse/KAFKA-3170
 Project: Kafka
  Issue Type: Bug
  Components: consumer
Affects Versions: 0.9.0.0
Reporter: Rajini Sivaram
Assignee: Rajini Sivaram


FETCH_MIN_BYTES_DOC says:

{quote}
The minimum amount of data the server should return for a fetch request. If 
insufficient data is available the request will wait for that much data to 
accumulate before answering the request. The default setting of 1 byte means 
that fetch requests are answered as soon as a single byte of data is available 
or the fetch request times out waiting for data to arrive. Setting this to 
something greater than 1 will cause the server to wait for larger amounts of 
data to accumulate which can improve server throughput a bit at the cost of 
some additional latency.
{quote}

But the default value is actually set to 1024. Either the doc or the value 
needs to be changed. Perhaps 1 is a better default?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3146) Stuck consumer with new consumer API in 0.9

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15123512#comment-15123512
 ] 

Ismael Juma commented on KAFKA-3146:


Thanks for reporting back [~darkjh]

> Stuck consumer with new consumer API in 0.9
> ---
>
> Key: KAFKA-3146
> URL: https://issues.apache.org/jira/browse/KAFKA-3146
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.9.0.0
>Reporter: Han JU
>Assignee: Jason Gustafson
>
> I'm prototyping with the new consumer API of kafka 0.9 and I'm particularly 
> interested in the `ConsumerRebalanceListener`.
> My test setup is like the following:
>   - 5M messages pre-loaded in one node kafka 0.9
>   - 12 partitions, auto offset commit set to false
>   - in `onPartitionsRevoked`, commit offset and flush the local state
> The test run is like the following:
>   - launch one process with 2 consumers and let it consume for a while
>   - launch another process with 2 consumers, this triggers a rebalancing, and 
> let these 2 processes run until messages are all consumed
> The code is here: https://gist.github.com/darkjh/fe1e5a5387bf13b4d4dd
> So at first, the 2 consumers of the first process each got 6 partitions. And 
> after the rebalancing, each consumer got 3 partitions. It's confirmed by 
> logging inside the `onPartitionAssigned` callback.
> But after the rebalancing, one of the 2 consumers of the first process stop 
> receiving messages, even if it has partitions assigned to: 
> balance-1 pulled 7237 msgs ...
> balance-0 pulled 7263 msgs ...
> 2016-01-22 15:50:37,533 [INFO] [pool-1-thread-2] 
> o.a.k.c.c.i.AbstractCoordinator - Attempt to heart beat failed since the 
> group is rebalancing, try to re-join group.
> balance-1 flush @ 536637
> balance-1 committed offset for List(balance-11, balance-10, balance-9, 
> balance-8, balance-7, balance-6)
> 2016-01-22 15:50:37,575 [INFO] [pool-1-thread-1] 
> o.a.k.c.c.i.AbstractCoordinator - Attempt to heart beat failed since the 
> group is rebalancing, try to re-join group.
> balance-0 flush @ 543845
> balance-0 committed offset for List(balance-5, balance-4, balance-3, 
> balance-2, balance-1, balance-0)
> balance-0 got assigned List(balance-5, balance-4, balance-3)
> balance-1 got assigned List(balance-11, balance-10, balance-9)
> balance-1 pulled 3625 msgs ...
> balance-0 pulled 3621 msgs ...
> balance-0 pulled 3631 msgs ...
> balance-0 pulled 3631 msgs ...
> balance-1 pulled 0 msgs ...
> balance-0 pulled 3643 msgs ...
> balance-0 pulled 3643 msgs ...
> balance-1 pulled 0 msgs ...
> balance-0 pulled 3622 msgs ...
> balance-0 pulled 3632 msgs ...
> balance-1 pulled 0 msgs ...
> balance-0 pulled 3637 msgs ...
> balance-0 pulled 3641 msgs ...
> balance-0 pulled 3640 msgs ...
> balance-1 pulled 0 msgs ...
> balance-0 pulled 3632 msgs ...
> balance-0 pulled 3630 msgs ...
> balance-1 pulled 0 msgs ...
> ..
> `balance-0` and `balance-1` are the names of the consumer thread. So after 
> the rebalancing, thread `balance-1` continues to poll but no message arrive, 
> given that it has got 3 partitions assigned to after the rebalancing.
> Finally other 3 consumers pulls all their partitions' message, the situation 
> is like 
> GROUP, TOPIC, PARTITION, CURRENT OFFSET, LOG END OFFSET, LAG, OWNER
> balance-test, balance, 9, 417467, 417467, 0, consumer-2_/127.0.0.1
> balance-test, balance, 10, 417467, 417467, 0, consumer-2_/127.0.0.1
> balance-test, balance, 11, 417467, 417467, 0, consumer-2_/127.0.0.1
> balance-test, balance, 6, 180269, 417467, 237198, consumer-2_/127.0.0.1
> balance-test, balance, 7, 180036, 417468, 237432, consumer-2_/127.0.0.1
> balance-test, balance, 8, 180197, 417467, 237270, consumer-2_/127.0.0.1
> balance-test, balance, 3, 417467, 417467, 0, consumer-1_/127.0.0.1
> balance-test, balance, 4, 417468, 417468, 0, consumer-1_/127.0.0.1
> balance-test, balance, 5, 417468, 417468, 0, consumer-1_/127.0.0.1
> balance-test, balance, 0, 417467, 417467, 0, consumer-1_/127.0.0.1
> balance-test, balance, 1, 417467, 417467, 0, consumer-1_/127.0.0.1
> balance-test, balance, 2, 417467, 417467, 0, consumer-1_/127.0.0.1 
> So you can see, partition [6, 7, 8] still has messages, but the consumer 
> can't pull them after the rebalancing. 
> I've tried 0.9.0.0 release, trunk and 0.9.0 branch, for both server/broker 
> and client.
> One workaround (by Bruno Rassaerts), is to do a manual seek to the current 
> position in the `onPartitionsAssigned` call back.
> The corresponding mailing list discussion is here: 
> http://mail-archives.apache.org/mod_mbox/kafka-users/201601.mbox/%3CCA%2BndhHok%3DemRceLuhwGHKwMCVQSmgTUeaxs-ycK-U2nLcc8Uhg%40mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3029) Make class org.apache.kafka.common.TopicPartition Serializable

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3029:
---
Reviewer: Ewen Cheslack-Postava
  Status: Patch Available  (was: Open)

> Make class org.apache.kafka.common.TopicPartition Serializable
> --
>
> Key: KAFKA-3029
> URL: https://issues.apache.org/jira/browse/KAFKA-3029
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 0.9.0.0
>Reporter: Praveen Devarao
>  Labels: easyfix, patch
> Fix For: 0.9.0.1
>
>
> The client class TopicPartition is exposed and used by consumer applications 
> directly. In case where the application needs to checkpoint the state it is 
> difficult to serialize different app classes that use TopicPartition as 
> TopicParitition is not serializable.
> For instance consider the Spark use case where RDDs have to be 
> checkpointedthe KafakaInputDstream (which we are currently modifying to 
> use the new Kafka API rather than the Highlevel apis in previous version) 
> cannot be serialized due to above limitation.
> I have created a patch to serialize TopicPartition class by making it 
> implement serializable interface and have issued a pull request.
> Can this be merged for the next release (0.9.0.1)
> Thanks
> Praveen



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-3146) Stuck consumer with new consumer API in 0.9

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-3146.

Resolution: Fixed

> Stuck consumer with new consumer API in 0.9
> ---
>
> Key: KAFKA-3146
> URL: https://issues.apache.org/jira/browse/KAFKA-3146
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.9.0.0
>Reporter: Han JU
>Assignee: Jason Gustafson
>
> I'm prototyping with the new consumer API of kafka 0.9 and I'm particularly 
> interested in the `ConsumerRebalanceListener`.
> My test setup is like the following:
>   - 5M messages pre-loaded in one node kafka 0.9
>   - 12 partitions, auto offset commit set to false
>   - in `onPartitionsRevoked`, commit offset and flush the local state
> The test run is like the following:
>   - launch one process with 2 consumers and let it consume for a while
>   - launch another process with 2 consumers, this triggers a rebalancing, and 
> let these 2 processes run until messages are all consumed
> The code is here: https://gist.github.com/darkjh/fe1e5a5387bf13b4d4dd
> So at first, the 2 consumers of the first process each got 6 partitions. And 
> after the rebalancing, each consumer got 3 partitions. It's confirmed by 
> logging inside the `onPartitionAssigned` callback.
> But after the rebalancing, one of the 2 consumers of the first process stop 
> receiving messages, even if it has partitions assigned to: 
> balance-1 pulled 7237 msgs ...
> balance-0 pulled 7263 msgs ...
> 2016-01-22 15:50:37,533 [INFO] [pool-1-thread-2] 
> o.a.k.c.c.i.AbstractCoordinator - Attempt to heart beat failed since the 
> group is rebalancing, try to re-join group.
> balance-1 flush @ 536637
> balance-1 committed offset for List(balance-11, balance-10, balance-9, 
> balance-8, balance-7, balance-6)
> 2016-01-22 15:50:37,575 [INFO] [pool-1-thread-1] 
> o.a.k.c.c.i.AbstractCoordinator - Attempt to heart beat failed since the 
> group is rebalancing, try to re-join group.
> balance-0 flush @ 543845
> balance-0 committed offset for List(balance-5, balance-4, balance-3, 
> balance-2, balance-1, balance-0)
> balance-0 got assigned List(balance-5, balance-4, balance-3)
> balance-1 got assigned List(balance-11, balance-10, balance-9)
> balance-1 pulled 3625 msgs ...
> balance-0 pulled 3621 msgs ...
> balance-0 pulled 3631 msgs ...
> balance-0 pulled 3631 msgs ...
> balance-1 pulled 0 msgs ...
> balance-0 pulled 3643 msgs ...
> balance-0 pulled 3643 msgs ...
> balance-1 pulled 0 msgs ...
> balance-0 pulled 3622 msgs ...
> balance-0 pulled 3632 msgs ...
> balance-1 pulled 0 msgs ...
> balance-0 pulled 3637 msgs ...
> balance-0 pulled 3641 msgs ...
> balance-0 pulled 3640 msgs ...
> balance-1 pulled 0 msgs ...
> balance-0 pulled 3632 msgs ...
> balance-0 pulled 3630 msgs ...
> balance-1 pulled 0 msgs ...
> ..
> `balance-0` and `balance-1` are the names of the consumer thread. So after 
> the rebalancing, thread `balance-1` continues to poll but no message arrive, 
> given that it has got 3 partitions assigned to after the rebalancing.
> Finally other 3 consumers pulls all their partitions' message, the situation 
> is like 
> GROUP, TOPIC, PARTITION, CURRENT OFFSET, LOG END OFFSET, LAG, OWNER
> balance-test, balance, 9, 417467, 417467, 0, consumer-2_/127.0.0.1
> balance-test, balance, 10, 417467, 417467, 0, consumer-2_/127.0.0.1
> balance-test, balance, 11, 417467, 417467, 0, consumer-2_/127.0.0.1
> balance-test, balance, 6, 180269, 417467, 237198, consumer-2_/127.0.0.1
> balance-test, balance, 7, 180036, 417468, 237432, consumer-2_/127.0.0.1
> balance-test, balance, 8, 180197, 417467, 237270, consumer-2_/127.0.0.1
> balance-test, balance, 3, 417467, 417467, 0, consumer-1_/127.0.0.1
> balance-test, balance, 4, 417468, 417468, 0, consumer-1_/127.0.0.1
> balance-test, balance, 5, 417468, 417468, 0, consumer-1_/127.0.0.1
> balance-test, balance, 0, 417467, 417467, 0, consumer-1_/127.0.0.1
> balance-test, balance, 1, 417467, 417467, 0, consumer-1_/127.0.0.1
> balance-test, balance, 2, 417467, 417467, 0, consumer-1_/127.0.0.1 
> So you can see, partition [6, 7, 8] still has messages, but the consumer 
> can't pull them after the rebalancing. 
> I've tried 0.9.0.0 release, trunk and 0.9.0 branch, for both server/broker 
> and client.
> One workaround (by Bruno Rassaerts), is to do a manual seek to the current 
> position in the `onPartitionsAssigned` call back.
> The corresponding mailing list discussion is here: 
> http://mail-archives.apache.org/mod_mbox/kafka-users/201601.mbox/%3CCA%2BndhHok%3DemRceLuhwGHKwMCVQSmgTUeaxs-ycK-U2nLcc8Uhg%40mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3003) The fetch.wait.max.ms is not honored when new log segment rolled for low volume topics.

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3003:
---
Reviewer: Guozhang Wang
  Status: Patch Available  (was: Open)

> The fetch.wait.max.ms is not honored when new log segment rolled for low 
> volume topics.
> ---
>
> Key: KAFKA-3003
> URL: https://issues.apache.org/jira/browse/KAFKA-3003
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.9.0.0
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.9.0.1
>
>
> The problem we saw can be explained by the example below:
> 1. Message offset 100 is appended to partition p0, log segment .log. 
> at time T. After that no message is appended. 
> 2. This message is replicated, leader replica update its 
> highWatermark.messageOffset=100, highWatermark.segmentBaseOffset=0.
> 3. At time T + retention.ms, because no message has been appended to current 
> active log segment for retention.ms, the last modified time of the current 
> log segment reaches retention time. 
> 4. Broker rolls out a new log segment 0001.log, and deletes the old log 
> segment .log. The new log segment in this case is empty because there 
> is no message appended. 
> 5. In Log, the nextOffsetMetadata.segmentBaseOffset will be updated to the 
> new log segment's base offset, but nextOffsetMetadata.messageOffset does not 
> change. so nextOffsetMetadata.messageOffset=1, 
> nextOffsetMetadata.segmentBaseOffset=1.
> 6. Now a FetchRequest comes and try to fetch from offset 1, 
> fetch.wait.max.ms=1000.
> 7. In ReplicaManager, because there is no data to return, the fetch request 
> will be put into purgatory. When delayedFetchPurgatory.tryCompleteElseWatch() 
> is called, the DelayedFetch.tryComplete() compares replica.highWatermark and 
> the fetchOffset returned by log.read(), it will see the 
> replica.highWatermark.segmentBaseOffset=0 and 
> fetchOffset.segmentBaseOffset=1. So it will assume the fetch occurs on a 
> later segment and complete the delayed fetch immediately.
> In this case, the replica.highWatermark was not updated because the 
> LogOffsetMetadata.preceds() only checks the messageOffset but ignored 
> segmentBaseOffset. The fix is to let LogOffsetMetadata first check the 
> messageOffset then check the segmentBaseOffset. So replica.highWatermark will 
> get updated after the follower fetches from the leader.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3170) Default value of fetch_min_bytes in new consumer is 1024 while doc says it is 1

[Rajini Sivaram (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajini Sivaram updated KAFKA-3170:
--
Status: Patch Available  (was: Open)

> Default value of fetch_min_bytes in new consumer is 1024 while doc says it is 
> 1
> ---
>
> Key: KAFKA-3170
> URL: https://issues.apache.org/jira/browse/KAFKA-3170
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.9.0.0
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
>
> FETCH_MIN_BYTES_DOC says:
> {quote}
> The minimum amount of data the server should return for a fetch request. If 
> insufficient data is available the request will wait for that much data to 
> accumulate before answering the request. The default setting of 1 byte means 
> that fetch requests are answered as soon as a single byte of data is 
> available or the fetch request times out waiting for data to arrive. Setting 
> this to something greater than 1 will cause the server to wait for larger 
> amounts of data to accumulate which can improve server throughput a bit at 
> the cost of some additional latency.
> {quote}
> But the default value is actually set to 1024. Either the doc or the value 
> needs to be changed. Perhaps 1 is a better default?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3170) Default value of fetch_min_bytes in new consumer is 1024 while doc says it is 1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15123499#comment-15123499
 ] 

ASF GitHub Bot commented on KAFKA-3170:
---

GitHub user rajinisivaram opened a pull request:

https://github.com/apache/kafka/pull/832

KAFKA-3170: Set default fetch_min_bytes in new consumer to 1

Set default to 1 instead of 1024, this matches the existing doc and feels 
like a better default value. Have run the unit tests with the change.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rajinisivaram/kafka KAFKA-3170

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/832.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #832


commit 7e9f944a61e331738ff7c9ed7ee93495039d697f
Author: Rajini Sivaram 
Date:   2016-01-29T13:47:08Z

KAFKA-3170: Set default fetch_min_bytes in new consumer to 1




> Default value of fetch_min_bytes in new consumer is 1024 while doc says it is 
> 1
> ---
>
> Key: KAFKA-3170
> URL: https://issues.apache.org/jira/browse/KAFKA-3170
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.9.0.0
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
>
> FETCH_MIN_BYTES_DOC says:
> {quote}
> The minimum amount of data the server should return for a fetch request. If 
> insufficient data is available the request will wait for that much data to 
> accumulate before answering the request. The default setting of 1 byte means 
> that fetch requests are answered as soon as a single byte of data is 
> available or the fetch request times out waiting for data to arrive. Setting 
> this to something greater than 1 will cause the server to wait for larger 
> amounts of data to accumulate which can improve server throughput a bit at 
> the cost of some additional latency.
> {quote}
> But the default value is actually set to 1024. Either the doc or the value 
> needs to be changed. Perhaps 1 is a better default?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka_0.9.0_jdk7 #102

[Apache Jenkins Server]

See 

Changes:

[harsha] KAFKA-3166; Disable SSL client authentication for SASL_SSL security

--
[...truncated 2834 lines...]
kafka.log.LogManagerTest > testCheckpointRecoveryPoints PASSED

kafka.log.LogManagerTest > testTimeBasedFlush PASSED

kafka.log.LogManagerTest > testCreateLog PASSED

kafka.log.LogManagerTest > testRecoveryDirectoryMappingWithTrailingSlash PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[0] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[1] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[2] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[3] PASSED

kafka.security.auth.ZkAuthorizationTest > testIsZkSecurityEnabled PASSED

kafka.security.auth.ZkAuthorizationTest > testZkUtils PASSED

kafka.security.auth.ZkAuthorizationTest > testZkAntiMigration PASSED

kafka.security.auth.ZkAuthorizationTest > testZkMigration PASSED

kafka.security.auth.ZkAuthorizationTest > testChroot PASSED

kafka.security.auth.ZkAuthorizationTest > testDelete PASSED

kafka.security.auth.ZkAuthorizationTest > testDeleteRecursive PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testAllowAllAccess PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testNoAclFound PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testAclManagementAPIs PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testWildCardAcls PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testTopicAcl PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testSuperUserHasAccess PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testDenyTakesPrecedence PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testNoAclFoundOverride PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testLoadCache PASSED

kafka.admin.AdminTest > testBasicPreferredReplicaElection PASSED

kafka.admin.AdminTest > testPreferredReplicaJsonData PASSED

kafka.admin.AdminTest > testReassigningNonExistingPartition PASSED

kafka.admin.AdminTest > testBootstrapClientIdConfig PASSED

kafka.admin.AdminTest > testPartitionReassignmentNonOverlappingReplicas PASSED

kafka.admin.AdminTest > testReplicaAssignment PASSED

kafka.admin.AdminTest > testPartitionReassignmentWithLeaderNotInNewReplicas 
PASSED

kafka.admin.AdminTest > testTopicConfigChange PASSED

kafka.admin.AdminTest > testResumePartitionReassignmentThatWasCompleted PASSED

kafka.admin.AdminTest > testManualReplicaAssignment PASSED

kafka.admin.AdminTest > testPartitionReassignmentWithLeaderInNewReplicas PASSED

kafka.admin.AdminTest > testShutdownBroker PASSED

kafka.admin.AdminTest > testTopicCreationWithCollision PASSED

kafka.admin.AdminTest > testTopicCreationInZK PASSED

kafka.admin.TopicCommandTest > testTopicDeletion PASSED

kafka.admin.TopicCommandTest > testConfigPreservationAcrossPartitionAlteration 
PASSED

kafka.admin.DeleteTopicTest > testDeleteTopicWithCleaner PASSED

kafka.admin.DeleteTopicTest > testResumeDeleteTopicOnControllerFailover PASSED

kafka.admin.DeleteTopicTest > testResumeDeleteTopicWithRecoveredFollower PASSED

kafka.admin.DeleteTopicTest > testDeleteTopicAlreadyMarkedAsDeleted PASSED

kafka.admin.DeleteTopicTest > testPartitionReassignmentDuringDeleteTopic PASSED

kafka.admin.DeleteTopicTest > testDeleteNonExistingTopic PASSED

kafka.admin.DeleteTopicTest > testRecreateTopicAfterDeletion PASSED

kafka.admin.DeleteTopicTest > testAddPartitionDuringDeleteTopic PASSED

kafka.admin.DeleteTopicTest > testDeleteTopicWithAllAliveReplicas PASSED

kafka.admin.DeleteTopicTest > testDeleteTopicDuringAddPartition PASSED

kafka.admin.DeleteConsumerGroupTest > 
testGroupWideDeleteInZKDoesNothingForActiveConsumerGroup PASSED

kafka.admin.DeleteConsumerGroupTest > 
testGroupTopicWideDeleteInZKDoesNothingForActiveGroupConsumingMultipleTopics 
PASSED

kafka.admin.DeleteConsumerGroupTest > 
testConsumptionOnRecreatedTopicAfterTopicWideDeleteInZK PASSED

kafka.admin.DeleteConsumerGroupTest > testTopicWideDeleteInZK PASSED

kafka.admin.DeleteConsumerGroupTest > 
testGroupTopicWideDeleteInZKForGroupConsumingOneTopic PASSED

kafka.admin.DeleteConsumerGroupTest > 
testGroupTopicWideDeleteInZKForGroupConsumingMultipleTopics PASSED

kafka.admin.DeleteConsumerGroupTest > testGroupWideDeleteInZK PASSED

kafka.admin.AddPartitionsTest > testWrongReplicaCount PASSED

kafka.admin.AddPartitionsTest > testTopicDoesNotExist PASSED

kafka.admin.AddPartitionsTest > testIncrementPartitions PASSED

kafka.admin.AddPartitionsTest > testManualAssignmentOfReplicas PASSED

kafka.admin.AddPartitionsTest > testReplicaPlacement PASSED

kafka.admin.ConfigCommandTest > testArgumentParse PASSED
:clients:checkstyleMain
FAILURE: Build failed with an exception.

* What went wrong:
Failed to capture snapshot of input files for task 'checkstyleMain' during 
up-to-date check.  See stacktrace for details.
> Could not read entry 
> 

[jira] [Updated] (KAFKA-3147) Memory records is not writable in MirrorMaker

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3147:
---
Fix Version/s: 0.9.0.1

> Memory records is not writable in MirrorMaker
> -
>
> Key: KAFKA-3147
> URL: https://issues.apache.org/jira/browse/KAFKA-3147
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.0
>Reporter: Meghana Narasimhan
>Assignee: Mayuresh Gharat
> Fix For: 0.9.0.1
>
>
> Hi,
> We are running a 3 node cluster (kafka version 0.9) and Node 0 also has a few 
> mirror makers running. 
> When we do a rolling restart of the cluster, the mirror maker shuts down with 
> the following errors.
> [2016-01-11 20:16:00,348] WARN Got error produce response with correlation id 
> 12491674 on topic-partition test-99, retrying (2147483646 attempts left). 
> Error: NOT_LEADER_FOR_PARTITION 
> (org.apache.kafka.clients.producer.internals.Sender)
> [2016-01-11 20:16:00,853] FATAL [mirrormaker-thread-0] Mirror maker thread 
> failure due to  (kafka.tools.MirrorMaker$MirrorMakerThread)
> java.lang.IllegalStateException: Memory records is not writable
> at 
> org.apache.kafka.common.record.MemoryRecords.append(MemoryRecords.java:93)
> at 
> org.apache.kafka.clients.producer.internals.RecordBatch.tryAppend(RecordBatch.java:69)
> at 
> org.apache.kafka.clients.producer.internals.RecordAccumulator.append(RecordAccumulator.java:168)
> at 
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:435)
> at 
> kafka.tools.MirrorMaker$MirrorMakerProducer.send(MirrorMaker.scala:593)
> at 
> kafka.tools.MirrorMaker$MirrorMakerThread$$anonfun$run$3.apply(MirrorMaker.scala:398)
> at 
> kafka.tools.MirrorMaker$MirrorMakerThread$$anonfun$run$3.apply(MirrorMaker.scala:398)
> at scala.collection.Iterator$class.foreach(Iterator.scala:742)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at 
> kafka.tools.MirrorMaker$MirrorMakerThread.run(MirrorMaker.scala:398)
> [2016-01-11 20:16:01,072] WARN Got error produce response with correlation id 
> 12491679 on topic-partition test-75, retrying (2147483646 attempts left). 
> Error: NETWORK_EXCEPTION (org.apache.kafka.clients.producer.internals.Sender)
> [2016-01-11 20:16:01,073] WARN Got error produce response with correlation id 
> 12491679 on topic-partition test-93, retrying (2147483646 attempts left). 
> Error: NETWORK_EXCEPTION (org.apache.kafka.clients.producer.internals.Sender)
> [2016-01-11 20:16:01,073] WARN Got error produce response with correlation id 
> 12491679 on topic-partition test-24, retrying (2147483646 attempts left). 
> Error: NETWORK_EXCEPTION (org.apache.kafka.clients.producer.internals.Sender)
> [2016-01-11 20:16:20,479] FATAL [mirrormaker-thread-0] Mirror maker thread 
> exited abnormally, stopping the whole mirror maker. 
> (kafka.tools.MirrorMaker$MirrorMakerThread)
> Curious if the NOT_LEADER_FOR_PARTITION is because of a potential bug hinted 
> at in the thread , 
> http://mail-archives.apache.org/mod_mbox/kafka-users/201505.mbox/%3ccajs3ho_u8s1xou_kudnfjamypjtmrjlw10qvkngn2yqkdan...@mail.gmail.com%3E
>
> And I think the mirror maker shuts down because of the 
> "abort.on.send.failure" which is set to true in our case. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1595) Remove deprecated and slower scala JSON parser from kafka.consumer.TopicCount

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15123214#comment-15123214
 ] 

ASF GitHub Bot commented on KAFKA-1595:
---

Github user ijuma closed the pull request at:

https://github.com/apache/kafka/pull/83


> Remove deprecated and slower scala JSON parser from kafka.consumer.TopicCount
> -
>
> Key: KAFKA-1595
> URL: https://issues.apache.org/jira/browse/KAFKA-1595
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.1.1
>Reporter: Jagbir
>Assignee: Ismael Juma
>  Labels: newbie
>
> The following issue is created as a follow up suggested by Jun Rao
> in a kafka news group message with the Subject
> "Blocking Recursive parsing from 
> kafka.consumer.TopicCount$.constructTopicCount"
> SUMMARY:
> An issue was detected in a typical cluster of 3 kafka instances backed
> by 3 zookeeper instances (kafka version 0.8.1.1, scala version 2.10.3,
> java version 1.7.0_65). On consumer end, when consumers get recycled,
> there is a troubling JSON parsing recursion which takes a busy lock and
> blocks consumers thread pool.
> In 0.8.1.1 scala client library ZookeeperConsumerConnector.scala:355 takes
> a global lock (0xd3a7e1d0) during the rebalance, and fires an
> expensive JSON parsing, while keeping the other consumers from shutting
> down, see, e.g,
> at 
> kafka.consumer.ZookeeperConsumerConnector.shutdown(ZookeeperConsumerConnector.scala:161)
> The deep recursive JSON parsing should be deprecated in favor
> of a better JSON parser, see, e.g,
> http://engineering.ooyala.com/blog/comparing-scala-json-libraries?
> DETAILS:
> The first dump is for a recursive blocking thread holding the lock for 
> 0xd3a7e1d0
> and the subsequent dump is for a waiting thread.
> (Please grep for 0xd3a7e1d0 to see the locked object.)
> Â 
> -8<-
> "Sa863f22b1e5hjh6788991800900b34545c_profile-a-prod1-s-140789080845312-c397945e8_watcher_executor"
> prio=10 tid=0x7f24dc285800 nid=0xda9 runnable [0x7f249e40b000]
> java.lang.Thread.State: RUNNABLE
> at 
> scala.util.parsing.combinator.Parsers$$anonfun$rep1$1.p$7(Parsers.scala:722)
> at 
> scala.util.parsing.combinator.Parsers$$anonfun$rep1$1.continue$1(Parsers.scala:726)
> at 
> scala.util.parsing.combinator.Parsers$$anonfun$rep1$1.apply(Parsers.scala:737)
> at 
> scala.util.parsing.combinator.Parsers$$anonfun$rep1$1.apply(Parsers.scala:721)
> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> at 
> scala.util.parsing.combinator.Parsers$Success.flatMapWithNext(Parsers.scala:142)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$flatMap$1.apply(Parsers.scala:239)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$flatMap$1.apply(Parsers.scala:239)
> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$flatMap$1.apply(Parsers.scala:239)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$flatMap$1.apply(Parsers.scala:239)
> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> at 

[GitHub] kafka pull request: KAFKA-1595; Remove deprecated and slower scala...

[ijuma]

Github user ijuma closed the pull request at:

https://github.com/apache/kafka/pull/83


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3146) Stuck consumer with new consumer API in 0.9

[Han JU (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15123430#comment-15123430
 ] 

Han JU commented on KAFKA-3146:
---

[~hachikuji] Thanks a lot Jason. I've used 0.9.0 branch for both broker and 
client code, and it indeed solved the problem, even without regularly commit 
offset.
Thanks also for the tips! I think this can be closed.  

> Stuck consumer with new consumer API in 0.9
> ---
>
> Key: KAFKA-3146
> URL: https://issues.apache.org/jira/browse/KAFKA-3146
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.9.0.0
>Reporter: Han JU
>Assignee: Jason Gustafson
>
> I'm prototyping with the new consumer API of kafka 0.9 and I'm particularly 
> interested in the `ConsumerRebalanceListener`.
> My test setup is like the following:
>   - 5M messages pre-loaded in one node kafka 0.9
>   - 12 partitions, auto offset commit set to false
>   - in `onPartitionsRevoked`, commit offset and flush the local state
> The test run is like the following:
>   - launch one process with 2 consumers and let it consume for a while
>   - launch another process with 2 consumers, this triggers a rebalancing, and 
> let these 2 processes run until messages are all consumed
> The code is here: https://gist.github.com/darkjh/fe1e5a5387bf13b4d4dd
> So at first, the 2 consumers of the first process each got 6 partitions. And 
> after the rebalancing, each consumer got 3 partitions. It's confirmed by 
> logging inside the `onPartitionAssigned` callback.
> But after the rebalancing, one of the 2 consumers of the first process stop 
> receiving messages, even if it has partitions assigned to: 
> balance-1 pulled 7237 msgs ...
> balance-0 pulled 7263 msgs ...
> 2016-01-22 15:50:37,533 [INFO] [pool-1-thread-2] 
> o.a.k.c.c.i.AbstractCoordinator - Attempt to heart beat failed since the 
> group is rebalancing, try to re-join group.
> balance-1 flush @ 536637
> balance-1 committed offset for List(balance-11, balance-10, balance-9, 
> balance-8, balance-7, balance-6)
> 2016-01-22 15:50:37,575 [INFO] [pool-1-thread-1] 
> o.a.k.c.c.i.AbstractCoordinator - Attempt to heart beat failed since the 
> group is rebalancing, try to re-join group.
> balance-0 flush @ 543845
> balance-0 committed offset for List(balance-5, balance-4, balance-3, 
> balance-2, balance-1, balance-0)
> balance-0 got assigned List(balance-5, balance-4, balance-3)
> balance-1 got assigned List(balance-11, balance-10, balance-9)
> balance-1 pulled 3625 msgs ...
> balance-0 pulled 3621 msgs ...
> balance-0 pulled 3631 msgs ...
> balance-0 pulled 3631 msgs ...
> balance-1 pulled 0 msgs ...
> balance-0 pulled 3643 msgs ...
> balance-0 pulled 3643 msgs ...
> balance-1 pulled 0 msgs ...
> balance-0 pulled 3622 msgs ...
> balance-0 pulled 3632 msgs ...
> balance-1 pulled 0 msgs ...
> balance-0 pulled 3637 msgs ...
> balance-0 pulled 3641 msgs ...
> balance-0 pulled 3640 msgs ...
> balance-1 pulled 0 msgs ...
> balance-0 pulled 3632 msgs ...
> balance-0 pulled 3630 msgs ...
> balance-1 pulled 0 msgs ...
> ..
> `balance-0` and `balance-1` are the names of the consumer thread. So after 
> the rebalancing, thread `balance-1` continues to poll but no message arrive, 
> given that it has got 3 partitions assigned to after the rebalancing.
> Finally other 3 consumers pulls all their partitions' message, the situation 
> is like 
> GROUP, TOPIC, PARTITION, CURRENT OFFSET, LOG END OFFSET, LAG, OWNER
> balance-test, balance, 9, 417467, 417467, 0, consumer-2_/127.0.0.1
> balance-test, balance, 10, 417467, 417467, 0, consumer-2_/127.0.0.1
> balance-test, balance, 11, 417467, 417467, 0, consumer-2_/127.0.0.1
> balance-test, balance, 6, 180269, 417467, 237198, consumer-2_/127.0.0.1
> balance-test, balance, 7, 180036, 417468, 237432, consumer-2_/127.0.0.1
> balance-test, balance, 8, 180197, 417467, 237270, consumer-2_/127.0.0.1
> balance-test, balance, 3, 417467, 417467, 0, consumer-1_/127.0.0.1
> balance-test, balance, 4, 417468, 417468, 0, consumer-1_/127.0.0.1
> balance-test, balance, 5, 417468, 417468, 0, consumer-1_/127.0.0.1
> balance-test, balance, 0, 417467, 417467, 0, consumer-1_/127.0.0.1
> balance-test, balance, 1, 417467, 417467, 0, consumer-1_/127.0.0.1
> balance-test, balance, 2, 417467, 417467, 0, consumer-1_/127.0.0.1 
> So you can see, partition [6, 7, 8] still has messages, but the consumer 
> can't pull them after the rebalancing. 
> I've tried 0.9.0.0 release, trunk and 0.9.0 branch, for both server/broker 
> and client.
> One workaround (by Bruno Rassaerts), is to do a manual seek to the current 
> position in the `onPartitionsAssigned` call back.
> The corresponding mailing list discussion is here: 
> 

[GitHub] kafka pull request: MINOR: getRootLogger() should be accessed in s...

[vesense]

Github user vesense closed the pull request at:

https://github.com/apache/kafka/pull/397


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3166) Disable SSL client authentication for SASL_SSL security protocol

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15123453#comment-15123453
 ] 

ASF GitHub Bot commented on KAFKA-3166:
---

Github user ijuma closed the pull request at:

https://github.com/apache/kafka/pull/830


> Disable SSL client authentication for SASL_SSL security protocol
> 
>
> Key: KAFKA-3166
> URL: https://issues.apache.org/jira/browse/KAFKA-3166
> Project: Kafka
>  Issue Type: Improvement
>  Components: security
>Affects Versions: 0.9.0.0
>Reporter: Ismael Juma
>Assignee: Ismael Juma
>
> A useful scenario is for a broker to require clients to authenticate either 
> via SSL or via SASL (with SASL_SSL security protocol). With the current code, 
> this is not possible to achieve. If we set `ssl.client.auth` to `required`, 
> then it will be required for both SSL and SASL.
> I suggest we hardcode `ssl.client.auth` to `none` for the `SASL_SSL` case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2676) Incorrect package name in tests (unit and integration prefix)

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-2676:
---
Description: AclCommandTest and a few others have the package unit.* or 
integration.*. We should remove the unit or integration prefix.  (was: 
AclCommandTest has the package unit.kafka.security.auth. We should remove the 
unit par. ResourceTypeTest has the same issue.)

> Incorrect package name in tests (unit and integration prefix)
> -
>
> Key: KAFKA-2676
> URL: https://issues.apache.org/jira/browse/KAFKA-2676
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: chen zhu
>  Labels: newbie
>
> AclCommandTest and a few others have the package unit.* or integration.*. We 
> should remove the unit or integration prefix.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2676) Incorrect package name in tests (unit and integration prefix)

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-2676:
---
Summary: Incorrect package name in tests (unit and integration prefix)  
(was: AclCommandTest has wrong package name )

> Incorrect package name in tests (unit and integration prefix)
> -
>
> Key: KAFKA-2676
> URL: https://issues.apache.org/jira/browse/KAFKA-2676
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: chen zhu
>  Labels: newbie
>
> AclCommandTest has the package unit.kafka.security.auth. We should remove the 
> unit par. ResourceTypeTest has the same issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-3169) Kafka broker throws OutOfMemory error with invalid SASL packet

[Rajini Sivaram (JIRA)]

Rajini Sivaram created KAFKA-3169:
-

 Summary: Kafka broker throws OutOfMemory error with invalid SASL 
packet
 Key: KAFKA-3169
 URL: https://issues.apache.org/jira/browse/KAFKA-3169
 Project: Kafka
  Issue Type: Bug
  Components: security
Affects Versions: 0.9.0.0
Reporter: Rajini Sivaram
Assignee: Rajini Sivaram
Priority: Critical


Receive buffer used in Kafka servers to process SASL packets is unbounded. This 
can results in brokers crashing with OutOfMemory error when an invalid SASL 
packet is received. 

There is a standard SASL property in Java _javax.security.sasl.maxbuffer_ that 
can be used to specify buffer size. When properties are added to the Sasl 
implementation in KAFKA-3149, we can use the standard property to limit receive 
buffer size. 

But since this is a potential DoS issue, we should set a reasonable limit in 
0.9.0.1. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk7 #1001

[Apache Jenkins Server]

See 

Changes:

[harsha] KAFKA-3166; Disable SSL client authentication for SASL_SSL security

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on ubuntu-4 (docker Ubuntu ubuntu4 ubuntu yahoo-not-h2) in 
workspace 
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/kafka.git # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/kafka.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision 91491986c0a4c9c4cde2ab33be822852b76759e6 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 91491986c0a4c9c4cde2ab33be822852b76759e6
 > git rev-list 962aec1a78cac18608556ac99ab583cdc944ac3f # timeout=10
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
Setting 
JDK_1_7U51_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk-1.7u51
[kafka-trunk-jdk7] $ /bin/bash -xe /tmp/hudson3490531291515366858.sh
+ 
/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2/bin/gradle
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
http://gradle.org/docs/2.4-rc-2/userguide/gradle_daemon.html.
Building project 'core' with Scala version 2.10.6
:downloadWrapper

BUILD SUCCESSFUL

Total time: 13.889 secs
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
Setting 
JDK_1_7U51_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk-1.7u51
[kafka-trunk-jdk7] $ /bin/bash -xe /tmp/hudson890868745849397099.sh
+ export GRADLE_OPTS=-Xmx1024m
+ GRADLE_OPTS=-Xmx1024m
+ ./gradlew -Dorg.gradle.project.maxParallelForks=1 clean jarAll testAll
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
https://docs.gradle.org/2.10/userguide/gradle_daemon.html.
Building project 'core' with Scala version 2.10.6
:clean UP-TO-DATE
:clients:clean UP-TO-DATE
:connect:clean UP-TO-DATE
:core:clean UP-TO-DATE
:examples:clean UP-TO-DATE
:log4j-appender:clean UP-TO-DATE
:streams:clean UP-TO-DATE
:tools:clean UP-TO-DATE
:connect:api:clean UP-TO-DATE
:connect:file:clean UP-TO-DATE
:connect:json:clean UP-TO-DATE
:connect:runtime:clean UP-TO-DATE
:streams:examples:clean UP-TO-DATE
:jar_core_2_10
Building project 'core' with Scala version 2.10.6
:kafka-trunk-jdk7:clients:compileJava
:jar_core_2_10 FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Failed to capture snapshot of input files for task 'compileJava' during 
up-to-date check.  See stacktrace for details.
> Could not add entry 
> '/home/jenkins/.gradle/caches/modules-2/files-2.1/net.jpountz.lz4/lz4/1.3.0/c708bb2590c0652a642236ef45d9f99ff842a2ce/lz4-1.3.0.jar'
>  to cache fileHashes.bin 
> (

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.

BUILD FAILED

Total time: 17.41 secs
Build step 'Execute shell' marked build as failure
Recording test results
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
Setting 
JDK_1_7U51_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk-1.7u51
ERROR: Publisher 'Publish JUnit test result report' failed: No test report 
files were found. Configuration error?
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
Setting 
JDK_1_7U51_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk-1.7u51

[jira] [Updated] (KAFKA-3169) Kafka broker throws OutOfMemory error with invalid SASL packet

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3169:
---
Fix Version/s: 0.9.0.1

> Kafka broker throws OutOfMemory error with invalid SASL packet
> --
>
> Key: KAFKA-3169
> URL: https://issues.apache.org/jira/browse/KAFKA-3169
> Project: Kafka
>  Issue Type: Bug
>  Components: security
>Affects Versions: 0.9.0.0
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
>Priority: Critical
> Fix For: 0.9.0.1
>
>
> Receive buffer used in Kafka servers to process SASL packets is unbounded. 
> This can results in brokers crashing with OutOfMemory error when an invalid 
> SASL packet is received. 
> There is a standard SASL property in Java _javax.security.sasl.maxbuffer_ 
> that can be used to specify buffer size. When properties are added to the 
> Sasl implementation in KAFKA-3149, we can use the standard property to limit 
> receive buffer size. 
> But since this is a potential DoS issue, we should set a reasonable limit in 
> 0.9.0.1. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-42: Add Producer and Consumer Interceptors

[Becket Qin]

Anna,

It is still not clear to me why we should expose CRC to end user.
Followings are my confusions.

1. Isn't the TopicPartition + Offset already uniquely identified a message?
It seems better than CRC no matter from summary point of view or auditing
point of view.

2. Currently CRC only has 4 bytes. So it will have collision when there are
more than ~4 billion messages. Take LinkedIn as an example, we have 1.3
trillion messages per day. So there will be at least a couple of hundreds
collision for each CRC code every day, whereas TopicPartition+Offset will
not have any collision.

3. CRC is calculated after all the fields have been filled in by producer,
including timestamp, attributes, etc. It might also get recomputed on
broker side. So if users only get CRC from record metadata. Without having
the entire message bytes, they may not be able to verify its correctness,
and the CRC could even be invalid if the broker ever overwritten any field
or did format conversion.

Thanks,

Jiangjie (Becket) Qin




On Thu, Jan 28, 2016 at 5:58 PM, Anna Povzner  wrote:

> On a second thought, yes, I think we should expose record size that
> represents application bytes. This is Becket's option #1.
>
> I updated the KIP wiki with new fields in RecordMetadata and
> ConsumerRecord.
>
> I would like to start a voting thread tomorrow if there are no objections
> or more things to discuss.
>
> Thanks,
> Anna
>
>
>
> On Thu, Jan 28, 2016 at 2:25 PM, Anna Povzner  wrote:
>
> > Regarding record size as bytes sent over the wire, my concern is that it
> > is almost impossible to calculate per-record. We could do as: 1)
> compressed
> > bytes / number of records in a compressed message, as Todd mentioned; or
> 2)
> > or same as #1 but take it proportional to uncompressed record size vs.
> > total uncompressed size of records. All of these calculations give us an
> > estimate. So maybe record size as bytes sent over the wire is not a
> > per-record metadata, but rather per topic/partition measure that is
> better
> > to be exposed through metrics?
> >
> >
> > On Thu, Jan 28, 2016 at 2:09 PM, Todd Palino  wrote:
> >
> >> It may be difficult (or nearly impossible) to get actual compressed
> bytes
> >> for a message from a compressed batch, but I do think it’s useful
> >> information to have available for the very reason noted, bandwidth
> >> consumed. Does it make sense to have an interceptor at the batch level
> >> that
> >> can provide this? The other option is to estimate it (such as making an
> >> assumption that the messages in a batch are equal in size, which is not
> >> necessarily true), which is probably not the right answer.
> >>
> >> -Todd
> >>
> >>
> >> On Thu, Jan 28, 2016 at 1:48 PM, Anna Povzner 
> wrote:
> >>
> >> > Hi Becket,
> >> >
> >> > It will be up to the interceptor to implement their audit or
> monitoring
> >> > strategy. I would also imagine there is more than one good way to do
> >> audit.
> >> > So, I agree that some of the interceptors may not use CRC, and we will
> >> not
> >> > require it. The question is now whether intercepting CRCs is needed. I
> >> > think they are very useful for monitoring and audit, because CRC
> >> provides
> >> > an a easy way to get a summary of a message, rather than using message
> >> > bytes or key/value objects.
> >> >
> >> > Regarding record size, I agree that bandwidth example was not a good
> >> one. I
> >> > think it would be hard to get actual bytes sent over the wire (your
> #2),
> >> > since multiple records get compressed together and we would need to
> >> decide
> >> > which bytes to account to which record. So I am inclined to only do
> your
> >> > #1. However, it still makes more sense to me just to return record
> size
> >> > including the header, since this is the actual record size.
> >> >
> >> > Thanks,
> >> > Anna
> >> >
> >> > On Thu, Jan 28, 2016 at 11:46 AM, Becket Qin 
> >> wrote:
> >> >
> >> > > Anna,
> >> > >
> >> > > Using CRC to do end2end auditing might be very costly because you
> will
> >> > need
> >> > > to collect all the CRC from both producer and consumer. And it is
> >> based
> >> > on
> >> > > the assumption that broker does not modify the record.
> >> > > Can you shed some idea on how end to end auditing will be using the
> >> CRC
> >> > > before we decide to expose such low level detail to the end user? It
> >> > would
> >> > > also be helpful if you can compare it with something like sequence
> >> number
> >> > > based auditing.
> >> > >
> >> > > About the record size, one thing worth notice is that the size of
> >> Record
> >> > is
> >> > > not the actual bytes sent over the wire if we use compression. So
> that
> >> > does
> >> > > not really tell user how much bandwidth they are using. Personally I
> >> > think
> >> > > two kinds of size may be useful.
> >> > > 1. The record size after serialization, i.e. application bytes. (The
> >> > 

Jenkins build is back to normal : kafka-trunk-jdk8 #327

[Apache Jenkins Server]

See 

[jira] [Updated] (KAFKA-3166) Disable SSL client authentication for SASL_SSL security protocol

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3166:
---
   Resolution: Fixed
 Reviewer: sriharsha chintalapani  (was: Jun Rao)
Fix Version/s: 0.9.0.1
   Status: Resolved  (was: Patch Available)

> Disable SSL client authentication for SASL_SSL security protocol
> 
>
> Key: KAFKA-3166
> URL: https://issues.apache.org/jira/browse/KAFKA-3166
> Project: Kafka
>  Issue Type: Improvement
>  Components: security
>Affects Versions: 0.9.0.0
>Reporter: Ismael Juma
>Assignee: Ismael Juma
> Fix For: 0.9.0.1
>
>
> A useful scenario is for a broker to require clients to authenticate either 
> via SSL or via SASL (with SASL_SSL security protocol). With the current code, 
> this is not possible to achieve. If we set `ssl.client.auth` to `required`, 
> then it will be required for both SSL and SASL.
> I suggest we hardcode `ssl.client.auth` to `none` for the `SASL_SSL` case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3169) Kafka broker throws OutOfMemory error with invalid SASL packet

[Rajini Sivaram (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajini Sivaram updated KAFKA-3169:
--
Status: Patch Available  (was: Open)

> Kafka broker throws OutOfMemory error with invalid SASL packet
> --
>
> Key: KAFKA-3169
> URL: https://issues.apache.org/jira/browse/KAFKA-3169
> Project: Kafka
>  Issue Type: Bug
>  Components: security
>Affects Versions: 0.9.0.0
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
>Priority: Critical
> Fix For: 0.9.0.1
>
>
> Receive buffer used in Kafka servers to process SASL packets is unbounded. 
> This can results in brokers crashing with OutOfMemory error when an invalid 
> SASL packet is received. 
> There is a standard SASL property in Java _javax.security.sasl.maxbuffer_ 
> that can be used to specify buffer size. When properties are added to the 
> Sasl implementation in KAFKA-3149, we can use the standard property to limit 
> receive buffer size. 
> But since this is a potential DoS issue, we should set a reasonable limit in 
> 0.9.0.1. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-43: Kafka SASL enhancements

[Rajini Sivaram]

Ismael,

The first packet from the client is deliberately empty to distinguish
between non-negotiating GSSAPI packet and a negotiation packet. If this
packet contained mechanisms, then the code in the broker to distinguish
this from a GSSAPI packet would be a bit messy. I was thinking that the
client needs to wait for server response anyway before it can select
a mechanism and start the actual SASL auth process. And once the
client gets the server response, it would send the selected mechanism
followed immediately by the first packet of the SASL auth. So perhaps the
overhead is not that bad. Did you have a different flow in mind?

On Fri, Jan 29, 2016 at 10:12 AM, Ismael Juma  wrote:

> Thanks Rajini. One question: would it make sense for the client to
> optimistically suggest its preferred SASL mechanism (or maybe mechanisms)
> to avoid a roundtrip?
>
> Ismael
>
> On Fri, Jan 29, 2016 at 10:04 AM, Rajini Sivaram <
> rajinisiva...@googlemail.com> wrote:
>
> > Following on from the KIP meeting on Tuesday, I have updated the KIP
> with a
> > flow for negotiation of mechanisms to support multiple SASL mechanisms
> > within a broker. I have also added a configurable Login interface to
> > support custom mechanisms which require ticket refresh - requested by Tao
> > Xiao.
> >
> > I will work on updating the PR in KAFKA-3149 over the next few days since
> > it will be useful for review.
> >
> > All comments and suggestions are welcome.
> >
> >
> > On Thu, Jan 28, 2016 at 2:35 PM, tao xiao  wrote:
> >
> > > Sounds like a good approach to add provider in login module. Would love
> > to
> > > see updates in the PR to reflect the changes in Login and
> > > AuthCallbackHandler.
> > >
> > > On Thu, 28 Jan 2016 at 19:31 Rajini Sivaram <
> > rajinisiva...@googlemail.com>
> > > wrote:
> > >
> > > > Tao,
> > > >
> > > > We currently add the security provider in a static initializer in our
> > > login
> > > > module. This ensures that the security provider is always installed
> > > before
> > > > Kafka creates SaslServer/SaslClient. As you say, it is also possible
> to
> > > > insert code into your application to add security provider before
> Kafka
> > > > clients are created. Since you can also configure the JDK to add new
> > > > security providers, I am not sure if there is value in adding more
> > > > configuration in Kafka to add security providers.
> > > >
> > > > On Thu, Jan 28, 2016 at 2:25 AM, tao xiao 
> > wrote:
> > > >
> > > > > The callback works for me as long as it has access to Subject and
> > > mechs.
> > > > > The other thing is how we can inject the customized security
> provider
> > > via
> > > > > Security.addProvider()? If I want to implement my own SASL mech I
> > need
> > > to
> > > > > call the addProvider() before SASL.create so that my own
> > implementation
> > > > of
> > > > > SASLClient/Sever can be returned. Any thoughts on this? we can
> either
> > > let
> > > > > users inject the provider in their logic code before creating a
> > > > > producer/consumer or Kafka does it for users
> > > > >
> > > > > On Thu, 28 Jan 2016 at 03:36 Rajini Sivaram <
> > > > rajinisiva...@googlemail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Tao,
> > > > > >
> > > > > > *javax.security.auth.callback.**CallbackHandler *is the standard
> > way
> > > in
> > > > > > which SASL clients and server obtain additional mechanism
> specific
> > > > > > input. *AuthCallbackHandler
> > > > > > *simply extends this interface to propagate configuration
> > > properties. I
> > > > > was
> > > > > > going to provide SASL mechanism and Subject to the callback
> > handlers
> > > as
> > > > > > well since the default handlers use these.
> > > > > >
> > > > > > Your SaslServer/SaslClient implementation can obtain the Subject
> > > using
> > > > > > *Subject.getSubject(**AccessController.getContext(). *But it will
> > be
> > > > good
> > > > > > to know if callback handlers would work for you - apart from
> > standard
> > > > > > callbacks like PasswordCallback, you can define your own
> callbacks
> > > too
> > > > if
> > > > > > you require.
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, Jan 27, 2016 at 3:59 PM, tao xiao 
> > > > wrote:
> > > > > >
> > > > > > > Thanks Rajini. The other thing in my mind is that we should
> find
> > a
> > > > way
> > > > > to
> > > > > > > expose subject to SASL so that other mechanisms are able to use
> > the
> > > > > > > principal and credentials stored in subject to do
> authentication.
> > > > > > >
> > > > > > > I am thinking to have below interface that can be extended by
> > users
> > > > to
> > > > > > > build the SASL client/server instead of having an AuthCallback.
> > > With
> > > > > this
> > > > > > > interface users are able to add their own security provider
> > before
> > > > > > > client/server is returned by SASL. Any thoughts?
> > > > > > >
> > > > > > > Interface SaslClientBuilder {

[jira] [Commented] (KAFKA-3166) Disable SSL client authentication for SASL_SSL security protocol

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15123323#comment-15123323
 ] 

ASF GitHub Bot commented on KAFKA-3166:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/827


> Disable SSL client authentication for SASL_SSL security protocol
> 
>
> Key: KAFKA-3166
> URL: https://issues.apache.org/jira/browse/KAFKA-3166
> Project: Kafka
>  Issue Type: Improvement
>  Components: security
>Affects Versions: 0.9.0.0
>Reporter: Ismael Juma
>Assignee: Ismael Juma
>
> A useful scenario is for a broker to require clients to authenticate either 
> via SSL or via SASL (with SASL_SSL security protocol). With the current code, 
> this is not possible to achieve. If we set `ssl.client.auth` to `required`, 
> then it will be required for both SSL and SASL.
> I suggest we hardcode `ssl.client.auth` to `none` for the `SASL_SSL` case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3166; Disable SSL client authentication ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/827


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: KAFKA-3166; Disable SSL client authentication ...

[ijuma]

Github user ijuma closed the pull request at:

https://github.com/apache/kafka/pull/830


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: KAFKA-3169: Limit receive buffer size for SASL...

[rajinisivaram]

GitHub user rajinisivaram opened a pull request:

https://github.com/apache/kafka/pull/831

KAFKA-3169: Limit receive buffer size for SASL packets in broker

Limit receive buffer size to avoid OOM in broker with invalid SASL packets

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rajinisivaram/kafka KAFKA-3169

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/831.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #831


commit add436b8d3ad03f1189547bb4bfac824295d7e63
Author: Rajini Sivaram 
Date:   2016-01-29T13:02:08Z

KAFKA-3169: Limit receive buffer size for SASL packets to avoid broker OOM 
with invalid packets




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3169) Kafka broker throws OutOfMemory error with invalid SASL packet

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15123458#comment-15123458
 ] 

ASF GitHub Bot commented on KAFKA-3169:
---

GitHub user rajinisivaram opened a pull request:

https://github.com/apache/kafka/pull/831

KAFKA-3169: Limit receive buffer size for SASL packets in broker

Limit receive buffer size to avoid OOM in broker with invalid SASL packets

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rajinisivaram/kafka KAFKA-3169

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/831.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #831


commit add436b8d3ad03f1189547bb4bfac824295d7e63
Author: Rajini Sivaram 
Date:   2016-01-29T13:02:08Z

KAFKA-3169: Limit receive buffer size for SASL packets to avoid broker OOM 
with invalid packets




> Kafka broker throws OutOfMemory error with invalid SASL packet
> --
>
> Key: KAFKA-3169
> URL: https://issues.apache.org/jira/browse/KAFKA-3169
> Project: Kafka
>  Issue Type: Bug
>  Components: security
>Affects Versions: 0.9.0.0
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
>Priority: Critical
> Fix For: 0.9.0.1
>
>
> Receive buffer used in Kafka servers to process SASL packets is unbounded. 
> This can results in brokers crashing with OutOfMemory error when an invalid 
> SASL packet is received. 
> There is a standard SASL property in Java _javax.security.sasl.maxbuffer_ 
> that can be used to specify buffer size. When properties are added to the 
> Sasl implementation in KAFKA-3149, we can use the standard property to limit 
> receive buffer size. 
> But since this is a potential DoS issue, we should set a reasonable limit in 
> 0.9.0.1. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3166; Disable SSL client authentication ...

[ijuma]

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/830

KAFKA-3166; Disable SSL client authentication for SASL_SSL security 
protocol (backport)

Backport of https://github.com/apache/kafka/pull/827 to 0.9.0 that only 
includes the essential code changes (excluded the test changes due to 
conflicts).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka 
kafka-3166-backport-disable-ssl-auth-sasl-ssl

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/830.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #830


commit b5513d6a1bb86c99aef81af40f63f24b005004ff
Author: Ismael Juma 
Date:   2016-01-28T19:51:14Z

SSL client authentication should be disabled for SASL_SSL security protocol




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Code Contribution wiki tweaks

[Ismael Juma]

Hi all,

I did a couple of tweaks to the "Contributing Code Changes"[1] wiki page:

1. I noted that PR submitters should add a comment to the PR after they've
addressed comments from reviewers (no notification is sent otherwise). I
added this because I noticed that some people weren't doing that and it
tends to cause the review process to take longer.

2. I changed the paragraph about dealing with conflicts to use merge
instead of rebase. Given that reviewers use the "Files changed" tab to
comment on the PR and that the merge script squashes commits before
merging, merging is better than rebasing because (1) it's easier for the
submitter and (2) it's easier for the reviewer to see what has changed
since the last review.

I think these changes are non-controversial (most people already do it this
way), but please do let me know if you disagree.

Ismael

[1]
https://cwiki.apache.org/confluence/display/KAFKA/Contributing+Code+Changes

Re: [DISCUSS] KIP-43: Kafka SASL enhancements

[tao xiao]

Hi Rajini,

Do you consider exposing Subject to AuthCallback as well? It is useful for
users building their own SASL mechanism so that we have control  where to
put logon data in subject and how to manipulate in SASL callback


On Fri, 29 Jan 2016 at 18:04 Rajini Sivaram 
wrote:

> Following on from the KIP meeting on Tuesday, I have updated the KIP with a
> flow for negotiation of mechanisms to support multiple SASL mechanisms
> within a broker. I have also added a configurable Login interface to
> support custom mechanisms which require ticket refresh - requested by Tao
> Xiao.
>
> I will work on updating the PR in KAFKA-3149 over the next few days since
> it will be useful for review.
>
> All comments and suggestions are welcome.
>
>
> On Thu, Jan 28, 2016 at 2:35 PM, tao xiao  wrote:
>
> > Sounds like a good approach to add provider in login module. Would love
> to
> > see updates in the PR to reflect the changes in Login and
> > AuthCallbackHandler.
> >
> > On Thu, 28 Jan 2016 at 19:31 Rajini Sivaram <
> rajinisiva...@googlemail.com>
> > wrote:
> >
> > > Tao,
> > >
> > > We currently add the security provider in a static initializer in our
> > login
> > > module. This ensures that the security provider is always installed
> > before
> > > Kafka creates SaslServer/SaslClient. As you say, it is also possible to
> > > insert code into your application to add security provider before Kafka
> > > clients are created. Since you can also configure the JDK to add new
> > > security providers, I am not sure if there is value in adding more
> > > configuration in Kafka to add security providers.
> > >
> > > On Thu, Jan 28, 2016 at 2:25 AM, tao xiao 
> wrote:
> > >
> > > > The callback works for me as long as it has access to Subject and
> > mechs.
> > > > The other thing is how we can inject the customized security provider
> > via
> > > > Security.addProvider()? If I want to implement my own SASL mech I
> need
> > to
> > > > call the addProvider() before SASL.create so that my own
> implementation
> > > of
> > > > SASLClient/Sever can be returned. Any thoughts on this? we can either
> > let
> > > > users inject the provider in their logic code before creating a
> > > > producer/consumer or Kafka does it for users
> > > >
> > > > On Thu, 28 Jan 2016 at 03:36 Rajini Sivaram <
> > > rajinisiva...@googlemail.com>
> > > > wrote:
> > > >
> > > > > Hi Tao,
> > > > >
> > > > > *javax.security.auth.callback.**CallbackHandler *is the standard
> way
> > in
> > > > > which SASL clients and server obtain additional mechanism specific
> > > > > input. *AuthCallbackHandler
> > > > > *simply extends this interface to propagate configuration
> > properties. I
> > > > was
> > > > > going to provide SASL mechanism and Subject to the callback
> handlers
> > as
> > > > > well since the default handlers use these.
> > > > >
> > > > > Your SaslServer/SaslClient implementation can obtain the Subject
> > using
> > > > > *Subject.getSubject(**AccessController.getContext(). *But it will
> be
> > > good
> > > > > to know if callback handlers would work for you - apart from
> standard
> > > > > callbacks like PasswordCallback, you can define your own callbacks
> > too
> > > if
> > > > > you require.
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Jan 27, 2016 at 3:59 PM, tao xiao 
> > > wrote:
> > > > >
> > > > > > Thanks Rajini. The other thing in my mind is that we should find
> a
> > > way
> > > > to
> > > > > > expose subject to SASL so that other mechanisms are able to use
> the
> > > > > > principal and credentials stored in subject to do authentication.
> > > > > >
> > > > > > I am thinking to have below interface that can be extended by
> users
> > > to
> > > > > > build the SASL client/server instead of having an AuthCallback.
> > With
> > > > this
> > > > > > interface users are able to add their own security provider
> before
> > > > > > client/server is returned by SASL. Any thoughts?
> > > > > >
> > > > > > Interface SaslClientBuilder {
> > > > > >
> > > > > > SaslClient build(mechs, subject, host, otherparams)
> > > > > > }
> > > > > >
> > > > > > Interface SaslServerBuilder {
> > > > > > SaslServer build(mechs, subject, host, otherparams)
> > > > > > }
> > > > > >
> > > > > > On Wed, 27 Jan 2016 at 18:54 Rajini Sivaram <
> > > > > rajinisiva...@googlemail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Tao,
> > > > > > >
> > > > > > > Thank you for the explanation. I couldn't find a standard Java
> > > > > interface
> > > > > > > that would be suitable, so will define one based on your
> > > requirement
> > > > > and
> > > > > > > update the KIP.
> > > > > > >
> > > > > > > Regards,
> > > > > > >
> > > > > > > Rajini
> > > > > > >
> > > > > > > On Wed, Jan 27, 2016 at 2:12 AM, tao xiao <
> xiaotao...@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > Hi Rajini,
> > > > > > > >
> > > > > > > > One requirement I 

[jira] [Commented] (KAFKA-3152) kafka-acl doesn't allow space in principal name

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15123233#comment-15123233
 ] 

ASF GitHub Bot commented on KAFKA-3152:
---

Github user ijuma closed the pull request at:

https://github.com/apache/kafka/pull/820


> kafka-acl doesn't allow space in principal name
> ---
>
> Key: KAFKA-3152
> URL: https://issues.apache.org/jira/browse/KAFKA-3152
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: Ismael Juma
> Fix For: 0.9.0.1
>
>
> When running the following,
> kafka-acls --authorizer kafka.security.auth.SimpleAclAuthorizer 
> --authorizer-properties zookeeper.connect=localhost:2181 --topic test --add 
> --producer --allow-host=* --allow-principal "User:CN=xxx,O=My Organization"
> the acl is set as the following. The part after space is ignored.
> Following is list of acls for resource: Topic:test 
>   User:CN=xxx,O=My has Allow permission for operations: Describe from 
> hosts: *
>   User:CN=xxx,O=My has Allow permission for operations: Write from hosts: 
> * 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-43: Kafka SASL enhancements

[Rajini Sivaram]

Following on from the KIP meeting on Tuesday, I have updated the KIP with a
flow for negotiation of mechanisms to support multiple SASL mechanisms
within a broker. I have also added a configurable Login interface to
support custom mechanisms which require ticket refresh - requested by Tao
Xiao.

I will work on updating the PR in KAFKA-3149 over the next few days since
it will be useful for review.

All comments and suggestions are welcome.


On Thu, Jan 28, 2016 at 2:35 PM, tao xiao  wrote:

> Sounds like a good approach to add provider in login module. Would love to
> see updates in the PR to reflect the changes in Login and
> AuthCallbackHandler.
>
> On Thu, 28 Jan 2016 at 19:31 Rajini Sivaram 
> wrote:
>
> > Tao,
> >
> > We currently add the security provider in a static initializer in our
> login
> > module. This ensures that the security provider is always installed
> before
> > Kafka creates SaslServer/SaslClient. As you say, it is also possible to
> > insert code into your application to add security provider before Kafka
> > clients are created. Since you can also configure the JDK to add new
> > security providers, I am not sure if there is value in adding more
> > configuration in Kafka to add security providers.
> >
> > On Thu, Jan 28, 2016 at 2:25 AM, tao xiao  wrote:
> >
> > > The callback works for me as long as it has access to Subject and
> mechs.
> > > The other thing is how we can inject the customized security provider
> via
> > > Security.addProvider()? If I want to implement my own SASL mech I need
> to
> > > call the addProvider() before SASL.create so that my own implementation
> > of
> > > SASLClient/Sever can be returned. Any thoughts on this? we can either
> let
> > > users inject the provider in their logic code before creating a
> > > producer/consumer or Kafka does it for users
> > >
> > > On Thu, 28 Jan 2016 at 03:36 Rajini Sivaram <
> > rajinisiva...@googlemail.com>
> > > wrote:
> > >
> > > > Hi Tao,
> > > >
> > > > *javax.security.auth.callback.**CallbackHandler *is the standard way
> in
> > > > which SASL clients and server obtain additional mechanism specific
> > > > input. *AuthCallbackHandler
> > > > *simply extends this interface to propagate configuration
> properties. I
> > > was
> > > > going to provide SASL mechanism and Subject to the callback handlers
> as
> > > > well since the default handlers use these.
> > > >
> > > > Your SaslServer/SaslClient implementation can obtain the Subject
> using
> > > > *Subject.getSubject(**AccessController.getContext(). *But it will be
> > good
> > > > to know if callback handlers would work for you - apart from standard
> > > > callbacks like PasswordCallback, you can define your own callbacks
> too
> > if
> > > > you require.
> > > >
> > > >
> > > >
> > > > On Wed, Jan 27, 2016 at 3:59 PM, tao xiao 
> > wrote:
> > > >
> > > > > Thanks Rajini. The other thing in my mind is that we should find a
> > way
> > > to
> > > > > expose subject to SASL so that other mechanisms are able to use the
> > > > > principal and credentials stored in subject to do authentication.
> > > > >
> > > > > I am thinking to have below interface that can be extended by users
> > to
> > > > > build the SASL client/server instead of having an AuthCallback.
> With
> > > this
> > > > > interface users are able to add their own security provider before
> > > > > client/server is returned by SASL. Any thoughts?
> > > > >
> > > > > Interface SaslClientBuilder {
> > > > >
> > > > > SaslClient build(mechs, subject, host, otherparams)
> > > > > }
> > > > >
> > > > > Interface SaslServerBuilder {
> > > > > SaslServer build(mechs, subject, host, otherparams)
> > > > > }
> > > > >
> > > > > On Wed, 27 Jan 2016 at 18:54 Rajini Sivaram <
> > > > rajinisiva...@googlemail.com>
> > > > > wrote:
> > > > >
> > > > > > Tao,
> > > > > >
> > > > > > Thank you for the explanation. I couldn't find a standard Java
> > > > interface
> > > > > > that would be suitable, so will define one based on your
> > requirement
> > > > and
> > > > > > update the KIP.
> > > > > >
> > > > > > Regards,
> > > > > >
> > > > > > Rajini
> > > > > >
> > > > > > On Wed, Jan 27, 2016 at 2:12 AM, tao xiao 
> > > > wrote:
> > > > > >
> > > > > > > Hi Rajini,
> > > > > > >
> > > > > > > One requirement I have is to refresh the login token every X
> > hours.
> > > > > Like
> > > > > > > what the Kerberos login does I need to have a background thread
> > > that
> > > > > > > refreshes the token periodically.
> > > > > > >
> > > > > > > I understand most of the login logic would be simple but it is
> > good
> > > > > that
> > > > > > we
> > > > > > > can expose the logic login to users and let them decide what
> they
> > > > want
> > > > > to
> > > > > > > do. And we can have a fallback login component that is used if
> > > users
> > > > > dont
> > > > > > > specify it.
> > > > > > >
> 

[jira] [Commented] (KAFKA-3166) Disable SSL client authentication for SASL_SSL security protocol

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15123282#comment-15123282
 ] 

ASF GitHub Bot commented on KAFKA-3166:
---

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/830

KAFKA-3166; Disable SSL client authentication for SASL_SSL security 
protocol (backport)

Backport of https://github.com/apache/kafka/pull/827 to 0.9.0 that only 
includes the essential code changes (excluded the test changes due to 
conflicts).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka 
kafka-3166-backport-disable-ssl-auth-sasl-ssl

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/830.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #830


commit b5513d6a1bb86c99aef81af40f63f24b005004ff
Author: Ismael Juma 
Date:   2016-01-28T19:51:14Z

SSL client authentication should be disabled for SASL_SSL security protocol




> Disable SSL client authentication for SASL_SSL security protocol
> 
>
> Key: KAFKA-3166
> URL: https://issues.apache.org/jira/browse/KAFKA-3166
> Project: Kafka
>  Issue Type: Improvement
>  Components: security
>Affects Versions: 0.9.0.0
>Reporter: Ismael Juma
>Assignee: Ismael Juma
>
> A useful scenario is for a broker to require clients to authenticate either 
> via SSL or via SASL (with SASL_SSL security protocol). With the current code, 
> this is not possible to achieve. If we set `ssl.client.auth` to `required`, 
> then it will be required for both SSL and SASL.
> I suggest we hardcode `ssl.client.auth` to `none` for the `SASL_SSL` case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-43: Kafka SASL enhancements

[Ismael Juma]

Thanks Rajini. One question: would it make sense for the client to
optimistically suggest its preferred SASL mechanism (or maybe mechanisms)
to avoid a roundtrip?

Ismael

On Fri, Jan 29, 2016 at 10:04 AM, Rajini Sivaram <
rajinisiva...@googlemail.com> wrote:

> Following on from the KIP meeting on Tuesday, I have updated the KIP with a
> flow for negotiation of mechanisms to support multiple SASL mechanisms
> within a broker. I have also added a configurable Login interface to
> support custom mechanisms which require ticket refresh - requested by Tao
> Xiao.
>
> I will work on updating the PR in KAFKA-3149 over the next few days since
> it will be useful for review.
>
> All comments and suggestions are welcome.
>
>
> On Thu, Jan 28, 2016 at 2:35 PM, tao xiao  wrote:
>
> > Sounds like a good approach to add provider in login module. Would love
> to
> > see updates in the PR to reflect the changes in Login and
> > AuthCallbackHandler.
> >
> > On Thu, 28 Jan 2016 at 19:31 Rajini Sivaram <
> rajinisiva...@googlemail.com>
> > wrote:
> >
> > > Tao,
> > >
> > > We currently add the security provider in a static initializer in our
> > login
> > > module. This ensures that the security provider is always installed
> > before
> > > Kafka creates SaslServer/SaslClient. As you say, it is also possible to
> > > insert code into your application to add security provider before Kafka
> > > clients are created. Since you can also configure the JDK to add new
> > > security providers, I am not sure if there is value in adding more
> > > configuration in Kafka to add security providers.
> > >
> > > On Thu, Jan 28, 2016 at 2:25 AM, tao xiao 
> wrote:
> > >
> > > > The callback works for me as long as it has access to Subject and
> > mechs.
> > > > The other thing is how we can inject the customized security provider
> > via
> > > > Security.addProvider()? If I want to implement my own SASL mech I
> need
> > to
> > > > call the addProvider() before SASL.create so that my own
> implementation
> > > of
> > > > SASLClient/Sever can be returned. Any thoughts on this? we can either
> > let
> > > > users inject the provider in their logic code before creating a
> > > > producer/consumer or Kafka does it for users
> > > >
> > > > On Thu, 28 Jan 2016 at 03:36 Rajini Sivaram <
> > > rajinisiva...@googlemail.com>
> > > > wrote:
> > > >
> > > > > Hi Tao,
> > > > >
> > > > > *javax.security.auth.callback.**CallbackHandler *is the standard
> way
> > in
> > > > > which SASL clients and server obtain additional mechanism specific
> > > > > input. *AuthCallbackHandler
> > > > > *simply extends this interface to propagate configuration
> > properties. I
> > > > was
> > > > > going to provide SASL mechanism and Subject to the callback
> handlers
> > as
> > > > > well since the default handlers use these.
> > > > >
> > > > > Your SaslServer/SaslClient implementation can obtain the Subject
> > using
> > > > > *Subject.getSubject(**AccessController.getContext(). *But it will
> be
> > > good
> > > > > to know if callback handlers would work for you - apart from
> standard
> > > > > callbacks like PasswordCallback, you can define your own callbacks
> > too
> > > if
> > > > > you require.
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Jan 27, 2016 at 3:59 PM, tao xiao 
> > > wrote:
> > > > >
> > > > > > Thanks Rajini. The other thing in my mind is that we should find
> a
> > > way
> > > > to
> > > > > > expose subject to SASL so that other mechanisms are able to use
> the
> > > > > > principal and credentials stored in subject to do authentication.
> > > > > >
> > > > > > I am thinking to have below interface that can be extended by
> users
> > > to
> > > > > > build the SASL client/server instead of having an AuthCallback.
> > With
> > > > this
> > > > > > interface users are able to add their own security provider
> before
> > > > > > client/server is returned by SASL. Any thoughts?
> > > > > >
> > > > > > Interface SaslClientBuilder {
> > > > > >
> > > > > > SaslClient build(mechs, subject, host, otherparams)
> > > > > > }
> > > > > >
> > > > > > Interface SaslServerBuilder {
> > > > > > SaslServer build(mechs, subject, host, otherparams)
> > > > > > }
> > > > > >
> > > > > > On Wed, 27 Jan 2016 at 18:54 Rajini Sivaram <
> > > > > rajinisiva...@googlemail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Tao,
> > > > > > >
> > > > > > > Thank you for the explanation. I couldn't find a standard Java
> > > > > interface
> > > > > > > that would be suitable, so will define one based on your
> > > requirement
> > > > > and
> > > > > > > update the KIP.
> > > > > > >
> > > > > > > Regards,
> > > > > > >
> > > > > > > Rajini
> > > > > > >
> > > > > > > On Wed, Jan 27, 2016 at 2:12 AM, tao xiao <
> xiaotao...@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > Hi Rajini,
> > > > > > > >
> > > > > > > > One requirement I have is to refresh the login token every X
> > > 

[jira] [Updated] (KAFKA-3167) Use local to the workspace Gradle cache and recreate it on every build

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3167:
---
Summary: Use local to the workspace Gradle cache and recreate it on every 
build  (was: Use local to the workspace cache and recreate it on every build)

> Use local to the workspace Gradle cache and recreate it on every build
> --
>
> Key: KAFKA-3167
> URL: https://issues.apache.org/jira/browse/KAFKA-3167
> Project: Kafka
>  Issue Type: Improvement
>  Components: build
>Reporter: Ismael Juma
>
> Kafka builds often fail with "Could not add entry 
> '/home/jenkins/.gradle/caches/modules-2/files-2.1/net.jpountz.lz4/lz4/1.3.0/c708bb2590c0652a642236ef45d9f99ff842a2ce/lz4-1.3.0.jar'
>  to cache fileHashes.bin"
> I filed INFRA-11083 and Andrew Bayer suggested:
> "Can you change your builds to use a local-to-the-workspace cache and then 
> nuke it/recreate it on every build?"
> This issue is about changing the Jenkins config for one of the trunk builds 
> to do the above to see if it helps.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3152; kafka-acl doesn't allow space in p...

[ijuma]

Github user ijuma closed the pull request at:

https://github.com/apache/kafka/pull/820


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-3171) min.insync.replicas not respected

[Victor Serbu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Victor Serbu updated KAFKA-3171:

Environment: Kafka downloaded from 
http://apache.javapipe.com/kafka/0.9.0.0/kafka_2.11-0.9.0.0.tgz

> min.insync.replicas not respected
> -
>
> Key: KAFKA-3171
> URL: https://issues.apache.org/jira/browse/KAFKA-3171
> Project: Kafka
>  Issue Type: Bug
> Environment: Kafka downloaded from 
> http://apache.javapipe.com/kafka/0.9.0.0/kafka_2.11-0.9.0.0.tgz
>Reporter: Victor Serbu
>
> Hello.
> I wanted to test kafka durability properties but I didn't managed to block 
> producer if number of ISR is less than min.insync.replicas.
> For my test I used 3 VM. On each VM is running kafka and zookeeper.
> Then I created a topic with min ISR = 2.
> {code}
> [root@kafka1 kafka_2.11-0.9.0.0]# bin/kafka-topics.sh --create --zookeeper 
> kafka1.novalocal:2181,kafka2.novalocal:2181,kafka3.novalocal:2181 
> --replication-factor 3 --partitions 1 --config 'min.insync.replicas=3' 
> --topic test61
> Created topic "test61".
> [root@kafka1 kafka_2.11-0.9.0.0]# bin/kafka-topics.sh --describe --zookeeper 
> kafka1.novalocal:2181,kafka2.novalocal:2181,kafka3.novalocal:2181 --topic 
> test60  Topic:test60
> PartitionCount:1ReplicationFactor:3 Configs:min.insync.replicas=2
> Topic: test60   Partition: 0Leader: 0   Replicas: 0,2,1 Isr: 
> 0,2,1
> {code}
> Then I stopped two of kafka brokers and started a producers and all messages 
> were commited (using a consumer I saw all the messages).
> {code}
> [root@kafka1 kafka_2.11-0.9.0.0]# bin/kafka-topics.sh --describe --zookeeper 
> kafka1.novalocal:2181,kafka2.novalocal:2181,kafka3.novalocal:2181 --topic 
> test60
> Topic:test60PartitionCount:1ReplicationFactor:3 
> Configs:min.insync.replicas=2
> Topic: test60   Partition: 0Leader: 0   Replicas: 0,2,1 Isr: 0
> {code}
> Producer
> {code}
> /**
>  * Licensed to the Apache Software Foundation (ASF) under one or more
>  * contributor license agreements.  See the NOTICE file distributed with
>  * this work for additional information regarding copyright ownership.
>  * The ASF licenses this file to You under the Apache License, Version 2.0
>  * (the "License"); you may not use this file except in compliance with
>  * the License.  You may obtain a copy of the License at
>  *
>  *http://www.apache.org/licenses/LICENSE-2.0
>  *
>  * Unless required by applicable law or agreed to in writing, software
>  * distributed under the License is distributed on an "AS IS" BASIS,
>  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>  * See the License for the specific language governing permissions and
>  * limitations under the License.
>  */
> package kafka.examples;
> import java.util.Properties;
> import java.util.Map;
> import java.util.Iterator;
> import java.util.concurrent.ExecutionException;
> import org.apache.kafka.clients.producer.Callback;
> import org.apache.kafka.clients.producer.KafkaProducer;
> import org.apache.kafka.clients.producer.ProducerRecord;
> import org.apache.kafka.clients.producer.RecordMetadata;
> import org.apache.kafka.common.Metric;
> import org.apache.kafka.common.MetricName;
> public class Producer extends Thread
> {
>   private final KafkaProducer producer;
>   private final String topic;
>   private final Boolean isAsync;
>   private final Integer maxId;
>   private final Boolean debug;
>   public Producer(String topic, Boolean isAsync, Integer maxId, Boolean debug)
>   {
> Properties props = new Properties();
> props.put("bootstrap.servers", 
> "kafka1.novalocal:9092,kafka2.novalocal:9092,kafka3.novalocal:9092");
> props.put("client.id", "DemoProducer");
> props.put("retries",1);
> props.put("key.serializer", 
> "org.apache.kafka.common.serialization.IntegerSerializer");
> props.put("value.serializer", 
> "org.apache.kafka.common.serialization.StringSerializer");
> props.put("ack", "all");
> producer = new KafkaProducer(props);
> this.topic = topic;
> this.isAsync = isAsync;
> this.maxId=maxId;
> this.debug = debug;
>   }
>   public void run() {
>   int messageNo = 1;
>   String messageStr;
>   long startTime = System.currentTimeMillis();
>   for (messageNo = 0; messageNo < this.maxId; messageNo++) {
>   messageStr = "{\"id\":" + messageNo + 
> ",\"msg_text\":\"aa\"}";
>   if (isAsync) { // Send asynchronously
> producer.send(new ProducerRecord(topic,
> messageNo,
> messageStr), new DemoCallBack(startTime, messageNo, 
> messageStr));
>   } else { // Send synchronously
>

[GitHub] kafka pull request: KAFKA-3086: Remove unused method.

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/758


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: KAFKA-3075; Fix ClassCastException in `Zookeep...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/833


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3075) java.util.HashMap cannot be cast to scala.collection.immutable.Map When using ZookeeperConsumerConnector.commitOffsets

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15123742#comment-15123742
 ] 

ASF GitHub Bot commented on KAFKA-3075:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/833


> java.util.HashMap cannot be cast to scala.collection.immutable.Map When using 
>  ZookeeperConsumerConnector.commitOffsets
> ---
>
> Key: KAFKA-3075
> URL: https://issues.apache.org/jira/browse/KAFKA-3075
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.9.0.0
>Reporter: Pengwei
>Assignee: Ismael Juma
> Fix For: 0.9.0.1
>
>
> When using java api's commit offset :
>  public void commitOffsets(Map 
> offsetsToCommit, boolean retryOnFailure);
> and pass a  Java Hash Map to this interface, will found:
> java.lang.ClassCastException: java.util.HashMap cannot be cast to 
> scala.collection.immutable.Map
> at 
> kafka.javaapi.consumer.ZookeeperConsumerConnector.commitOffsets(ZookeeperConsumerConnector.scala:118)
> at 
> kafka.examples.CommitExceptionTest.testCommitNotExistTopicShoudThrowException(CommitExceptionTest.java:55)
> at kafka.examples.CommitExceptionTest.main(CommitExceptionTest.java:20)
> Test case testCommitNotExistTopicShoudThrowException OK.
> Exception in thread "main" java.lang.ClassCastException: java.util.HashMap 
> cannot be cast to scala.collection.immutable.Map
> at 
> kafka.javaapi.consumer.ZookeeperConsumerConnector.commitOffsets(ZookeeperConsumerConnector.scala:118)
> at 
> kafka.examples.CommitExceptionTest.testCommitOffsetOutOfRange(CommitExceptionTest.java:95)
> at kafka.examples.CommitExceptionTest.main(CommitExceptionTest.java:22)
> The Origin Code:
>  def commitOffsets(offsetsToCommit: java.util.Map[TopicAndPartition, 
> OffsetAndMetadata], retryOnFailure: Boolean) {
> 
> underlying.commitOffsets(offsetsToCommit.asInstanceOf[immutable.Map[TopicAndPartition,
>  OffsetAndMetadata]], retryOnFailure)
>   }
> I try to fix like this, it is OK:
>   def commitOffsets(offsetsToCommit: java.util.Map[TopicAndPartition, 
> OffsetAndMetadata], retryOnFailure: Boolean) {
> import scala.collection.JavaConverters._
> underlying.commitOffsets(offsetsToCommit.asScala.toMap, retryOnFailure)
>   }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2936) Socket server selector can stuck on one send in tight loop.

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15123656#comment-15123656
 ] 

Ismael Juma commented on KAFKA-2936:


[~onurkaraman], are you still planning to submit a PR for this for 0.9.0.1?

> Socket server selector can stuck on one send in tight loop.
> ---
>
> Key: KAFKA-2936
> URL: https://issues.apache.org/jira/browse/KAFKA-2936
> Project: Kafka
>  Issue Type: Bug
>  Components: network
>Affects Versions: 0.9.0.0
>Reporter: Jiangjie Qin
>Assignee: Onur Karaman
> Fix For: 0.9.0.1
>
>
> When broker was sending a FetchResponse it is possible that the data to be 
> sent back is truncated. In this case, a KafkaException will be thrown. This 
> exception is caught by processor and the selector will be sending the message 
> in a tight while loop. It will continue doing this until the socket is closed 
> by the client due to request timeout.
> We should have a max retry for the response send.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2923) Improve 0.9.0 Upgrade Documents

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15123650#comment-15123650
 ] 

Ismael Juma commented on KAFKA-2923:


Point 1 was already done.

> Improve 0.9.0 Upgrade Documents 
> 
>
> Key: KAFKA-2923
> URL: https://issues.apache.org/jira/browse/KAFKA-2923
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>  Labels: newbie
> Fix For: 0.9.0.1
>
>
> A couple of places we can improve the upgrade docs:
> 1) Explanation about replica.lag.time.max.ms and how it relates to the old 
> configs.
> 2) Default quota configs.
> 3) Client-server compatibility: old clients working with new servers and new 
> clients working with old servers?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3075) java.util.HashMap cannot be cast to scala.collection.immutable.Map When using ZookeeperConsumerConnector.commitOffsets

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3075:
---
Affects Version/s: 0.9.0.0
   Status: Patch Available  (was: Open)

Thanks for the report.

> java.util.HashMap cannot be cast to scala.collection.immutable.Map When using 
>  ZookeeperConsumerConnector.commitOffsets
> ---
>
> Key: KAFKA-3075
> URL: https://issues.apache.org/jira/browse/KAFKA-3075
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.9.0.0
>Reporter: Pengwei
>Assignee: Ismael Juma
> Fix For: 0.9.0.1
>
>
> When using java api's commit offset :
>  public void commitOffsets(Map 
> offsetsToCommit, boolean retryOnFailure);
> and pass a  Java Hash Map to this interface, will found:
> java.lang.ClassCastException: java.util.HashMap cannot be cast to 
> scala.collection.immutable.Map
> at 
> kafka.javaapi.consumer.ZookeeperConsumerConnector.commitOffsets(ZookeeperConsumerConnector.scala:118)
> at 
> kafka.examples.CommitExceptionTest.testCommitNotExistTopicShoudThrowException(CommitExceptionTest.java:55)
> at kafka.examples.CommitExceptionTest.main(CommitExceptionTest.java:20)
> Test case testCommitNotExistTopicShoudThrowException OK.
> Exception in thread "main" java.lang.ClassCastException: java.util.HashMap 
> cannot be cast to scala.collection.immutable.Map
> at 
> kafka.javaapi.consumer.ZookeeperConsumerConnector.commitOffsets(ZookeeperConsumerConnector.scala:118)
> at 
> kafka.examples.CommitExceptionTest.testCommitOffsetOutOfRange(CommitExceptionTest.java:95)
> at kafka.examples.CommitExceptionTest.main(CommitExceptionTest.java:22)
> The Origin Code:
>  def commitOffsets(offsetsToCommit: java.util.Map[TopicAndPartition, 
> OffsetAndMetadata], retryOnFailure: Boolean) {
> 
> underlying.commitOffsets(offsetsToCommit.asInstanceOf[immutable.Map[TopicAndPartition,
>  OffsetAndMetadata]], retryOnFailure)
>   }
> I try to fix like this, it is OK:
>   def commitOffsets(offsetsToCommit: java.util.Map[TopicAndPartition, 
> OffsetAndMetadata], retryOnFailure: Boolean) {
> import scala.collection.JavaConverters._
> underlying.commitOffsets(offsetsToCommit.asScala.toMap, retryOnFailure)
>   }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3161: Fixed ProducerConfig/ConsumerConfi...

[crhyne]

Github user crhyne closed the pull request at:

https://github.com/apache/kafka/pull/824


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: KAFKA-3161: Fixed ProducerConfig/ConsumerConfi...

[crhyne]

GitHub user crhyne reopened a pull request:

https://github.com/apache/kafka/pull/824

KAFKA-3161: Fixed ProducerConfig/ConsumerConfig so that defaults are used 
in java.util.Properties

This impacts the ProducerConfig and ConsumerConfig.  I added a unit test to 
reflect the new case.  While running unit tests I found that the order for 
assertEquals(, ) were backwards - I fixed this.

This is my original work to be licensed to the kafka product (Instructions 
5d).


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/crhyne/kafka trunk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/824.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #824


commit 5c46a2711b5927b91d3076cd35675c1dd5eb6ca3
Author: Cliff Rhyne 
Date:   2016-01-28T16:02:08Z

cleaned up use of java.util.Properties so that defaults are used 
https://issues.apache.org/jira/browse/KAFKA-3161




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-1464) Add a throttling option to the Kafka replication tool

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15123618#comment-15123618
 ] 

Ismael Juma commented on KAFKA-1464:


Thanks for your input [~jkreps].

With regards to the issue where a replica may never catch up, it is a good 
point that came up previously. One option may be to disable throttling (or 
increase the catch-up rate) in the case where the replica is falling further 
behind.

One important question is whether users have enough information to be able to 
configure an appropriate throttling/catch-up rate that takes into account both 
disk IO and network bandwidth while keeping resource utilisation at an 
appropriate level. Thoughts? (the log cleaner has a similar config: 
`log.cleaner.io.max.bytes.per.second`, although it seems simpler to figure out).

> Add a throttling option to the Kafka replication tool
> -
>
> Key: KAFKA-1464
> URL: https://issues.apache.org/jira/browse/KAFKA-1464
> Project: Kafka
>  Issue Type: New Feature
>  Components: replication
>Affects Versions: 0.8.0
>Reporter: mjuarez
>Assignee: Ismael Juma
>Priority: Minor
>  Labels: replication, replication-tools
> Fix For: 0.9.1.0
>
>
> When performing replication on new nodes of a Kafka cluster, the replication 
> process will use all available resources to replicate as fast as possible.  
> This causes performance issues (mostly disk IO and sometimes network 
> bandwidth) when doing this in a production environment, in which you're 
> trying to serve downstream applications, at the same time you're performing 
> maintenance on the Kafka cluster.
> An option to throttle the replication to a specific rate (in either MB/s or 
> activities/second) would help production systems to better handle maintenance 
> tasks while still serving downstream applications.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

KAFKA-3002 issue in 0.9.0.0

[Roman Hoc]

Hello,
can you please provide approximate dates of releases, where this bug fix is
officially released?
I.e. either 0.9.0.1 or 0.9.1.0?
Thank you, best regards,
Roman

https://issues.apache.org/jira/browse/KAFKA-3002

[jira] [Created] (KAFKA-3171) min.insync.replicas not respected

[Victor Serbu (JIRA)]

Victor Serbu created KAFKA-3171:
---

 Summary: min.insync.replicas not respected
 Key: KAFKA-3171
 URL: https://issues.apache.org/jira/browse/KAFKA-3171
 Project: Kafka
  Issue Type: Bug
Reporter: Victor Serbu


Hello.

I wanted to test kafka durability properties but I didn't managed to block 
producer if number of ISR is less than min.insync.replicas.

For my test I used 3 VM. On each VM is running kafka and zookeeper.
Then I created a topic with min ISR = 2.
{code}
[root@kafka1 kafka_2.11-0.9.0.0]# bin/kafka-topics.sh --create --zookeeper 
kafka1.novalocal:2181,kafka2.novalocal:2181,kafka3.novalocal:2181 
--replication-factor 3 --partitions 1 --config 'min.insync.replicas=3' --topic 
test61
Created topic "test61".
[root@kafka1 kafka_2.11-0.9.0.0]# bin/kafka-topics.sh --describe --zookeeper 
kafka1.novalocal:2181,kafka2.novalocal:2181,kafka3.novalocal:2181 --topic 
test60  Topic:test60
PartitionCount:1ReplicationFactor:3 Configs:min.insync.replicas=2
Topic: test60   Partition: 0Leader: 0   Replicas: 0,2,1 Isr: 
0,2,1
{code}
Then I stopped two of kafka brokers and started a producers and all messages 
were commited (using a consumer I saw all the messages).
{code}
[root@kafka1 kafka_2.11-0.9.0.0]# bin/kafka-topics.sh --describe --zookeeper 
kafka1.novalocal:2181,kafka2.novalocal:2181,kafka3.novalocal:2181 --topic test60
Topic:test60PartitionCount:1ReplicationFactor:3 
Configs:min.insync.replicas=2
Topic: test60   Partition: 0Leader: 0   Replicas: 0,2,1 Isr: 0
{code}

Producer
{code}
/**
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package kafka.examples;


import java.util.Properties;
import java.util.Map;
import java.util.Iterator;
import java.util.concurrent.ExecutionException;

import org.apache.kafka.clients.producer.Callback;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.RecordMetadata;
import org.apache.kafka.common.Metric;
import org.apache.kafka.common.MetricName;

public class Producer extends Thread
{
  private final KafkaProducer producer;
  private final String topic;
  private final Boolean isAsync;
  private final Integer maxId;
  private final Boolean debug;

  public Producer(String topic, Boolean isAsync, Integer maxId, Boolean debug)
  {
Properties props = new Properties();
props.put("bootstrap.servers", 
"kafka1.novalocal:9092,kafka2.novalocal:9092,kafka3.novalocal:9092");
props.put("client.id", "DemoProducer");
props.put("retries",1);
props.put("key.serializer", 
"org.apache.kafka.common.serialization.IntegerSerializer");
props.put("value.serializer", 
"org.apache.kafka.common.serialization.StringSerializer");
props.put("ack", "all");
producer = new KafkaProducer(props);
this.topic = topic;
this.isAsync = isAsync;
this.maxId=maxId;
this.debug = debug;
  }

  public void run() {
  int messageNo = 1;
  String messageStr;
  long startTime = System.currentTimeMillis();
  for (messageNo = 0; messageNo < this.maxId; messageNo++) {
  messageStr = "{\"id\":" + messageNo + ",\"msg_text\":\"aa\"}";
  if (isAsync) { // Send asynchronously
producer.send(new ProducerRecord(topic,
messageNo,
messageStr), new DemoCallBack(startTime, messageNo, 
messageStr));
  } else { // Send synchronously
try {
  if (this.debug) {
System.out.println("" + (int) (System.currentTimeMillis() / 
1000L) + " Send message: (" + messageNo + ", " + messageStr + ")");
  }
  producer.send(new ProducerRecord(topic,
  messageNo,
  messageStr)).get();
  if (this.debug) {
System.out.println("" + (int) (System.currentTimeMillis() / 
1000L) + " Sent message: (" + messageNo + ", " + 

[jira] [Assigned] (KAFKA-3075) java.util.HashMap cannot be cast to scala.collection.immutable.Map When using ZookeeperConsumerConnector.commitOffsets

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma reassigned KAFKA-3075:
--

Assignee: Ismael Juma  (was: Neha Narkhede)

> java.util.HashMap cannot be cast to scala.collection.immutable.Map When using 
>  ZookeeperConsumerConnector.commitOffsets
> ---
>
> Key: KAFKA-3075
> URL: https://issues.apache.org/jira/browse/KAFKA-3075
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Reporter: Pengwei
>Assignee: Ismael Juma
> Fix For: 0.9.0.1
>
>
> When using java api's commit offset :
>  public void commitOffsets(Map 
> offsetsToCommit, boolean retryOnFailure);
> and pass a  Java Hash Map to this interface, will found:
> java.lang.ClassCastException: java.util.HashMap cannot be cast to 
> scala.collection.immutable.Map
> at 
> kafka.javaapi.consumer.ZookeeperConsumerConnector.commitOffsets(ZookeeperConsumerConnector.scala:118)
> at 
> kafka.examples.CommitExceptionTest.testCommitNotExistTopicShoudThrowException(CommitExceptionTest.java:55)
> at kafka.examples.CommitExceptionTest.main(CommitExceptionTest.java:20)
> Test case testCommitNotExistTopicShoudThrowException OK.
> Exception in thread "main" java.lang.ClassCastException: java.util.HashMap 
> cannot be cast to scala.collection.immutable.Map
> at 
> kafka.javaapi.consumer.ZookeeperConsumerConnector.commitOffsets(ZookeeperConsumerConnector.scala:118)
> at 
> kafka.examples.CommitExceptionTest.testCommitOffsetOutOfRange(CommitExceptionTest.java:95)
> at kafka.examples.CommitExceptionTest.main(CommitExceptionTest.java:22)
> The Origin Code:
>  def commitOffsets(offsetsToCommit: java.util.Map[TopicAndPartition, 
> OffsetAndMetadata], retryOnFailure: Boolean) {
> 
> underlying.commitOffsets(offsetsToCommit.asInstanceOf[immutable.Map[TopicAndPartition,
>  OffsetAndMetadata]], retryOnFailure)
>   }
> I try to fix like this, it is OK:
>   def commitOffsets(offsetsToCommit: java.util.Map[TopicAndPartition, 
> OffsetAndMetadata], retryOnFailure: Boolean) {
> import scala.collection.JavaConverters._
> underlying.commitOffsets(offsetsToCommit.asScala.toMap, retryOnFailure)
>   }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3068) NetworkClient may connect to a different Kafka cluster than originally configured

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3068:
---
Fix Version/s: 0.9.0.1

> NetworkClient may connect to a different Kafka cluster than originally 
> configured
> -
>
> Key: KAFKA-3068
> URL: https://issues.apache.org/jira/browse/KAFKA-3068
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: Eno Thereska
> Fix For: 0.9.0.1
>
>
> In https://github.com/apache/kafka/pull/290, we added the logic to cache all 
> brokers (id and ip) that the client has ever seen. If we can't find an 
> available broker from the current Metadata, we will pick a broker that we 
> have ever seen (in NetworkClient.leastLoadedNode()).
> One potential problem this logic can introduce is the following. Suppose that 
> we have a broker with id 1 in a Kafka cluster. A producer client remembers 
> this broker in nodesEverSeen. At some point, we bring down this broker and 
> use the host in a different Kafka cluster. Then, the producer client uses 
> this broker from nodesEverSeen to refresh metadata. It will find the metadata 
> in a different Kafka cluster and start producing data there.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3068) NetworkClient may connect to a different Kafka cluster than originally configured

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3068:
---
Reviewer: Ewen Cheslack-Postava
  Status: Patch Available  (was: Open)

> NetworkClient may connect to a different Kafka cluster than originally 
> configured
> -
>
> Key: KAFKA-3068
> URL: https://issues.apache.org/jira/browse/KAFKA-3068
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: Eno Thereska
> Fix For: 0.9.0.1
>
>
> In https://github.com/apache/kafka/pull/290, we added the logic to cache all 
> brokers (id and ip) that the client has ever seen. If we can't find an 
> available broker from the current Metadata, we will pick a broker that we 
> have ever seen (in NetworkClient.leastLoadedNode()).
> One potential problem this logic can introduce is the following. Suppose that 
> we have a broker with id 1 in a Kafka cluster. A producer client remembers 
> this broker in nodesEverSeen. At some point, we bring down this broker and 
> use the host in a different Kafka cluster. Then, the producer client uses 
> this broker from nodesEverSeen to refresh metadata. It will find the metadata 
> in a different Kafka cluster and start producing data there.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3075; Fix ClassCastException in `Zookeep...

[ijuma]

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/833

KAFKA-3075; Fix ClassCastException in 
`ZookeeperConsumerConnector.commitOffsets`



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka 
kafka-3075-javaapi-consumer-class-cast-exception

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/833.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #833


commit 2391fcb2cce2f4abcf9c4084c4f2cd45e1b0a88b
Author: Ismael Juma 
Date:   2016-01-29T16:32:51Z

Fix ClassCastException in `ZookeeperConsumerConnector.commitOffsets`




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3075) java.util.HashMap cannot be cast to scala.collection.immutable.Map When using ZookeeperConsumerConnector.commitOffsets

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15123710#comment-15123710
 ] 

ASF GitHub Bot commented on KAFKA-3075:
---

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/833

KAFKA-3075; Fix ClassCastException in 
`ZookeeperConsumerConnector.commitOffsets`



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka 
kafka-3075-javaapi-consumer-class-cast-exception

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/833.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #833


commit 2391fcb2cce2f4abcf9c4084c4f2cd45e1b0a88b
Author: Ismael Juma 
Date:   2016-01-29T16:32:51Z

Fix ClassCastException in `ZookeeperConsumerConnector.commitOffsets`




> java.util.HashMap cannot be cast to scala.collection.immutable.Map When using 
>  ZookeeperConsumerConnector.commitOffsets
> ---
>
> Key: KAFKA-3075
> URL: https://issues.apache.org/jira/browse/KAFKA-3075
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Reporter: Pengwei
>Assignee: Ismael Juma
> Fix For: 0.9.0.1
>
>
> When using java api's commit offset :
>  public void commitOffsets(Map 
> offsetsToCommit, boolean retryOnFailure);
> and pass a  Java Hash Map to this interface, will found:
> java.lang.ClassCastException: java.util.HashMap cannot be cast to 
> scala.collection.immutable.Map
> at 
> kafka.javaapi.consumer.ZookeeperConsumerConnector.commitOffsets(ZookeeperConsumerConnector.scala:118)
> at 
> kafka.examples.CommitExceptionTest.testCommitNotExistTopicShoudThrowException(CommitExceptionTest.java:55)
> at kafka.examples.CommitExceptionTest.main(CommitExceptionTest.java:20)
> Test case testCommitNotExistTopicShoudThrowException OK.
> Exception in thread "main" java.lang.ClassCastException: java.util.HashMap 
> cannot be cast to scala.collection.immutable.Map
> at 
> kafka.javaapi.consumer.ZookeeperConsumerConnector.commitOffsets(ZookeeperConsumerConnector.scala:118)
> at 
> kafka.examples.CommitExceptionTest.testCommitOffsetOutOfRange(CommitExceptionTest.java:95)
> at kafka.examples.CommitExceptionTest.main(CommitExceptionTest.java:22)
> The Origin Code:
>  def commitOffsets(offsetsToCommit: java.util.Map[TopicAndPartition, 
> OffsetAndMetadata], retryOnFailure: Boolean) {
> 
> underlying.commitOffsets(offsetsToCommit.asInstanceOf[immutable.Map[TopicAndPartition,
>  OffsetAndMetadata]], retryOnFailure)
>   }
> I try to fix like this, it is OK:
>   def commitOffsets(offsetsToCommit: java.util.Map[TopicAndPartition, 
> OffsetAndMetadata], retryOnFailure: Boolean) {
> import scala.collection.JavaConverters._
> underlying.commitOffsets(offsetsToCommit.asScala.toMap, retryOnFailure)
>   }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3075) java.util.HashMap cannot be cast to scala.collection.immutable.Map When using ZookeeperConsumerConnector.commitOffsets

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15123723#comment-15123723
 ] 

Ismael Juma commented on KAFKA-3075:


Not related to the bug report, but you may consider using the new Java consumer 
if possible:

http://www.confluent.io/blog/tutorial-getting-started-with-the-new-apache-kafka-0.9-consumer-client

> java.util.HashMap cannot be cast to scala.collection.immutable.Map When using 
>  ZookeeperConsumerConnector.commitOffsets
> ---
>
> Key: KAFKA-3075
> URL: https://issues.apache.org/jira/browse/KAFKA-3075
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.9.0.0
>Reporter: Pengwei
>Assignee: Ismael Juma
> Fix For: 0.9.0.1
>
>
> When using java api's commit offset :
>  public void commitOffsets(Map 
> offsetsToCommit, boolean retryOnFailure);
> and pass a  Java Hash Map to this interface, will found:
> java.lang.ClassCastException: java.util.HashMap cannot be cast to 
> scala.collection.immutable.Map
> at 
> kafka.javaapi.consumer.ZookeeperConsumerConnector.commitOffsets(ZookeeperConsumerConnector.scala:118)
> at 
> kafka.examples.CommitExceptionTest.testCommitNotExistTopicShoudThrowException(CommitExceptionTest.java:55)
> at kafka.examples.CommitExceptionTest.main(CommitExceptionTest.java:20)
> Test case testCommitNotExistTopicShoudThrowException OK.
> Exception in thread "main" java.lang.ClassCastException: java.util.HashMap 
> cannot be cast to scala.collection.immutable.Map
> at 
> kafka.javaapi.consumer.ZookeeperConsumerConnector.commitOffsets(ZookeeperConsumerConnector.scala:118)
> at 
> kafka.examples.CommitExceptionTest.testCommitOffsetOutOfRange(CommitExceptionTest.java:95)
> at kafka.examples.CommitExceptionTest.main(CommitExceptionTest.java:22)
> The Origin Code:
>  def commitOffsets(offsetsToCommit: java.util.Map[TopicAndPartition, 
> OffsetAndMetadata], retryOnFailure: Boolean) {
> 
> underlying.commitOffsets(offsetsToCommit.asInstanceOf[immutable.Map[TopicAndPartition,
>  OffsetAndMetadata]], retryOnFailure)
>   }
> I try to fix like this, it is OK:
>   def commitOffsets(offsetsToCommit: java.util.Map[TopicAndPartition, 
> OffsetAndMetadata], retryOnFailure: Boolean) {
> import scala.collection.JavaConverters._
> underlying.commitOffsets(offsetsToCommit.asScala.toMap, retryOnFailure)
>   }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3161) Refactor Java client's use of the Properties class

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15123758#comment-15123758
 ] 

ASF GitHub Bot commented on KAFKA-3161:
---

GitHub user crhyne reopened a pull request:

https://github.com/apache/kafka/pull/824

KAFKA-3161: Fixed ProducerConfig/ConsumerConfig so that defaults are used 
in java.util.Properties

This impacts the ProducerConfig and ConsumerConfig.  I added a unit test to 
reflect the new case.  While running unit tests I found that the order for 
assertEquals(, ) were backwards - I fixed this.

This is my original work to be licensed to the kafka product (Instructions 
5d).


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/crhyne/kafka trunk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/824.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #824


commit 5c46a2711b5927b91d3076cd35675c1dd5eb6ca3
Author: Cliff Rhyne 
Date:   2016-01-28T16:02:08Z

cleaned up use of java.util.Properties so that defaults are used 
https://issues.apache.org/jira/browse/KAFKA-3161




> Refactor Java client's use of the Properties class
> --
>
> Key: KAFKA-3161
> URL: https://issues.apache.org/jira/browse/KAFKA-3161
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 0.9.0.0
>Reporter: Cliff Rhyne
>
> The KafkaConsumer takes a Properties class for the config, but then instead 
> of using it's getProperty() function the class gets copied (which breaks the 
> use of defaults).
> One example is this from ConsumerConfig:
> Properties newProperties = new Properties();
> newProperties.putAll(properties);
> Which could be re-written as:
> Properties newProperties = new Properties(properties);
> This is important because applications using the client library expect to be 
> able to specify the default properties above.
> (I'm not sure how to go about this, but I'm working on the change locally 
> right now.  I'd like to assign it to myself but I guess I can't because i'm 
> not on the contributor list).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3161) Refactor Java client's use of the Properties class

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15123757#comment-15123757
 ] 

ASF GitHub Bot commented on KAFKA-3161:
---

Github user crhyne closed the pull request at:

https://github.com/apache/kafka/pull/824


> Refactor Java client's use of the Properties class
> --
>
> Key: KAFKA-3161
> URL: https://issues.apache.org/jira/browse/KAFKA-3161
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 0.9.0.0
>Reporter: Cliff Rhyne
>
> The KafkaConsumer takes a Properties class for the config, but then instead 
> of using it's getProperty() function the class gets copied (which breaks the 
> use of defaults).
> One example is this from ConsumerConfig:
> Properties newProperties = new Properties();
> newProperties.putAll(properties);
> Which could be re-written as:
> Properties newProperties = new Properties(properties);
> This is important because applications using the client library expect to be 
> able to specify the default properties above.
> (I'm not sure how to go about this, but I'm working on the change locally 
> right now.  I'd like to assign it to myself but I guess I can't because i'm 
> not on the contributor list).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2718) Reuse of temporary directories leading to transient unit test failures

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-2718:
---
Fix Version/s: (was: 0.9.0.1)
   0.9.1.0

> Reuse of temporary directories leading to transient unit test failures
> --
>
> Key: KAFKA-2718
> URL: https://issues.apache.org/jira/browse/KAFKA-2718
> Project: Kafka
>  Issue Type: Sub-task
>  Components: core
>Affects Versions: 0.9.1.0
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
> Fix For: 0.9.1.0
>
>
> Stack traces in some of the transient unit test failures indicate that 
> temporary directories used for Zookeeper are being reused.
> {quote}
> kafka.common.TopicExistsException: Topic "topic" already exists.
>   at 
> kafka.admin.AdminUtils$.createOrUpdateTopicPartitionAssignmentPathInZK(AdminUtils.scala:253)
>   at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:237)
>   at kafka.utils.TestUtils$.createTopic(TestUtils.scala:231)
>   at kafka.api.BaseConsumerTest.setUp(BaseConsumerTest.scala:63)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: MINOR: Fix spelling and grammar issues in Repl...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/829


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: kafka-consumer-groups.sh doesn't work when consumers are off

[Guozhang Wang]

Tao,

You are right, ConsumerOffsetChecker can still get offsets from the offset
manager in Kafka.

Guozhang

On Thu, Jan 28, 2016 at 9:36 PM, tao xiao  wrote:

> it first issues an offsetrequest to broker and check if offset is stored in
> broker if not it will queries zk
>
>
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala#L171
>
> On Fri, 29 Jan 2016 at 13:11 Guozhang Wang  wrote:
>
> > Tao,
> >
> > Hmm that is a bit wired since ConsumerOffsetChecker itself does not talk
> to
> > brokers at all, but only through ZK.
> >
> > Guozhang
> >
> > On Thu, Jan 28, 2016 at 6:07 PM, tao xiao  wrote:
> >
> > > Guozhang,
> > >
> > > The old ConsumerOffsetChecker works for new consumer too with offset
> > stored
> > > in Kafka. I tested it with mirror maker with new consumer enabled. it
> is
> > > able to show offset during mirror maker running and after its shutdown.
> > >
> > > On Fri, 29 Jan 2016 at 06:34 Guozhang Wang  wrote:
> > >
> > > > Once the offset is written to the log it is persistent and hence
> should
> > > > survive broker failures. And its retention policy is configurable.
> > > >
> > > > It may be a bit misleading in saying "in-memory cache" in my previous
> > > > email: the brokers just keep the in-memory map of [group, partition]
> ->
> > > > latest_offset, while the offset commit history is kept in the log.
> When
> > > we
> > > > delete the group, we remove the corresponding entry from memory map
> and
> > > put
> > > > a tombstone into log as well so that the old offsets will be
> compacted
> > > > eventually according to compaction policy.
> > > >
> > > > The old ConsumerOffsetChecker only works for old consumer that stores
> > > > offset in ZK.
> > > >
> > > > Guozhang
> > > >
> > > > On Thu, Jan 28, 2016 at 1:43 PM, Cliff Rhyne 
> wrote:
> > > >
> > > > > Hi Guozhang,
> > > > >
> > > > > That looks like it might help but feels like there might be some
> > gaps.
> > > > > Would it be able to survive restarts of the kafka broker?  How long
> > > would
> > > > > it stay in the cache (and is that configurable)?  If it expires
> from
> > > the
> > > > > cache, what's the cache-miss operation look like?  (yes, a lot of
> > this
> > > > > depends on the data still being in the logs to recover)
> > > > >
> > > > > In the mean time, can I rely on the deprecated
> ConsumerOffsetChecker
> > > > (which
> > > > > looks at zookeeper) even though I'm using the new KafkaConsumer?
> > > > >
> > > > > Thanks,
> > > > > Cliff
> > > > >
> > > > > On Thu, Jan 28, 2016 at 3:30 PM, Guozhang Wang  >
> > > > wrote:
> > > > >
> > > > > > Hi Cliff,
> > > > > >
> > > > > > Short answer to your question is it is just the current
> > > implementation.
> > > > > >
> > > > > > The coordinator stores the offsets as messages in an internal
> topic
> > > and
> > > > > > also keeps the latest offset values in in-memory. It answers
> > > > > > ConsumerGroupRequest using its cached offset, and upon the
> consumer
> > > > group
> > > > > > being removed since no member is alive already, it removed it
> from
> > > its
> > > > > > in-memory cache and add a "tombstone" to the offset log as well.
> > But
> > > > the
> > > > > > offsets are still persistent as messages in the log, which will
> > only
> > > be
> > > > > > compacted after a while (this is depend on the log compaction
> > > policy).
> > > > > >
> > > > > > There is a ticket open for improving on this scenario (
> > > > > > https://issues.apache.org/jira/browse/KAFKA-2720) which lets the
> > > > > > coordinator to only "purge" dead groups periodically instead of
> > > > > > immediately, and that may partially resolve your case.
> > > > > >
> > > > > > Guozhang
> > > > > >
> > > > > >
> > > > > > On Thu, Jan 28, 2016 at 12:13 PM, Cliff Rhyne 
> > > > wrote:
> > > > > >
> > > > > > > Just following up on this concern.  Is there a constraint that
> > > > prevents
> > > > > > > ConsumerGroupCommand from reporting offsets on a group if no
> > > members
> > > > > are
> > > > > > > connected, or is this just the current implementation?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Cliff
> > > > > > >
> > > > > > > On Mon, Jan 25, 2016 at 3:50 PM, Cliff Rhyne  >
> > > > wrote:
> > > > > > >
> > > > > > > > I'm running into a few challenges trying to evaluate offsets
> > and
> > > > lag
> > > > > > > > (pending message count) in the new Java KafkaConsumer.  The
> old
> > > > > > > > ConsumerOffsetChecker doesn't work anymore since the offsets
> > > aren't
> > > > > > > stored
> > > > > > > > in zookeeper after switching from the old consumer.  This
> would
> > > be
> > > > > > fine,
> > > > > > > > but the kafka-consumer-groups.sh command doesn't work if the
> > > > > consumers
> > > > > > > are
> > > > > > > > shut off.  This seems like an unnecessary limitation 

[jira] [Updated] (KAFKA-2758) Improve Offset Commit Behavior

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-2758:
---
Fix Version/s: (was: 0.9.0.1)
   0.9.1.0

> Improve Offset Commit Behavior
> --
>
> Key: KAFKA-2758
> URL: https://issues.apache.org/jira/browse/KAFKA-2758
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Guozhang Wang
>  Labels: newbiee
> Fix For: 0.9.1.0
>
>
> There are two scenarios of offset committing that we can improve:
> 1) we can filter the partitions whose committed offset is equal to the 
> consumed offset, meaning there is no new consumed messages from this 
> partition and hence we do not need to include this partition in the commit 
> request.
> 2) we can make a commit request right after resetting to a fetch / consume 
> position either according to the reset policy (e.g. on consumer starting up, 
> or handling of out of range offset, etc), or through the {code} seek {code} 
> so that if the consumer fails right after these event, upon recovery it can 
> restarts from the reset position instead of resetting again: this can lead 
> to, for example, data loss if we use "largest" as reset policy while there 
> are new messages coming to the fetching partitions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-42: Add Producer and Consumer Interceptors

[Joel Koshy]

Responding to some of the earlier comments in the thread:

@Jay/@Neha,

I think any one of onCommit/onAppend/onArrival would work for the concrete
use-case that I had outlined. I think onArrival is additionally useful for
custom validation - i.e., reject the message and do not append if it
violates some cluster-specific rule (for e.g., if some header timestamp is
older than xyz). However, the thing with user-supplied validation is we
would have to do with a (new) generic error code in the producer response.
While there is a risk of a broker interceptor having high latency I think
that is acceptable since it is the user's responsibility to ensure low
latency - the producer call-back and onAcknowledgment interceptor are
similar in this regard although those are less risky. Even so, I think
there are clear use-cases for broker interceptors so I feel the risk part
is something that just needs to be documented. @Jay that is a good point
about moving from Message/MessageSet to Records although that may be less
difficult to absorb since it is a broker-side interceptor and so people
don't need to get a ton of applications in their company to switch to use
it.

Re: onEnqueued: monitoring serialization latency can be done via metrics
but this is more useful for recording whether serialization succeeded or
not. onAcknowledgment subsumes this but it also subsumes other possible
errors (such as produce errors). It is more fine-grained than most people
need though (i.e., I don't think we will use it even if it is present.)

Re: checksums: I think it is a good addition to metadata; and for
low-volume or super-critical topics can be used for very strict auditing.

There are a couple of typos/edits for the wiki itself:

   - Under Kafka Producer changes:
   - you have references to KafkaConsumer constructor and
  ConsumerConfig.originals.
  - sendRecord -> sentRecord (may be clearer)
   - Under ProducerInterceptor interface: there is a mention of onEnqueued
   which was rejected
   - Comment for ConsumerRecord.record should probably be: // NEW: record
   size in bytes (*after decompression*)


BTW - Anna, nice work on the KIP!

Joel

On Fri, Jan 29, 2016 at 6:57 AM, Neha Narkhede  wrote:

> Becket,
>
> Is your concern the presence of CRC in the RecordMetadata or do you want to
> brainstorm how CRC can be used for auditing? I think we shouldn't try to
> think about the various ways that people can do monitoring using
> interceptors and the metadata we provide. The entire point of having
> pluggable interceptors is so that people can employ their own creative
> mechanisms to make use of interceptors.
>
> I do think that it is worth discussing whether or not CRC makes sense as
> record metadata to the user. My take is that the CRC is the best size-bound
> summary of serialized record content available to us which is expensive to
> recompute if the user were to redo it. I'd argue this summary of a record
> qualifies as its metadata. After all, we use the record CRC for a very
> important test of the system durability as it travels through the system.
>
> 1. Isn't the TopicPartition + Offset already uniquely identified a message?
> > It seems better than CRC no matter from summary point of view or auditing
> > point of view.
>
>
> The offset is a system-assigned value of uniqueness to the message. If you
> trusted the system that much, you are not looking to monitor it out-of-band
> :-)
>
>
> > 2. Currently CRC only has 4 bytes. So it will have collision when there
> are
> > more than ~4 billion messages. Take LinkedIn as an example, we have 1.3
> > trillion messages per day. So there will be at least a couple of hundreds
> > collision for each CRC code every day, whereas TopicPartition+Offset will
> > not have any collision.
>
>
> The CRC isn't sent over the wire and doesn't add any extra overhead in
> processing, so what is your concern? If you aren't convinced about its
> usefulness, you can always use the default do-nothing interceptor at
> LinkedIn and ignore the CRC.
>
> Without having
> > the entire message bytes, they may not be able to verify its correctness,
> > and the CRC could even be invalid if the broker ever overwritten any
> field
> > or did format conversion.
> >
>
> This doesn't make sense to me. The CRC is used for the most important
> durability check by Kafka - to verify that the message was not garbled over
> the wire. The system can't change it; it has to match on the consumer side
> or we will not return it to the user.
>
> On Fri, Jan 29, 2016 at 3:23 AM, Becket Qin  wrote:
>
> > Anna,
> >
> > It is still not clear to me why we should expose CRC to end user.
> > Followings are my confusions.
> >
> > 1. Isn't the TopicPartition + Offset already uniquely identified a
> message?
> > It seems better than CRC no matter from summary point of view or auditing
> > point of view.
> >
> > 2. Currently CRC only has 4 bytes. So it will have collision when there
> 

[jira] [Commented] (KAFKA-3086) unused handle method in MirrorMakerMessageHandler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15123729#comment-15123729
 ] 

ASF GitHub Bot commented on KAFKA-3086:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/758


> unused handle method in MirrorMakerMessageHandler
> -
>
> Key: KAFKA-3086
> URL: https://issues.apache.org/jira/browse/KAFKA-3086
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: Jakub Nowak
>  Labels: newbie
> Fix For: 0.9.1.0
>
>
> The following method is never used by MirrorMaker.
>   trait MirrorMakerMessageHandler {
> def handle(record: MessageAndMetadata[Array[Byte], Array[Byte]]): 
> util.List[ProducerRecord[Array[Byte], Array[Byte]]]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3086) unused handle method in MirrorMakerMessageHandler

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-3086:
-
   Resolution: Fixed
Fix Version/s: 0.9.1.0
   Status: Resolved  (was: Patch Available)

Issue resolved by pull request 758
[https://github.com/apache/kafka/pull/758]

> unused handle method in MirrorMakerMessageHandler
> -
>
> Key: KAFKA-3086
> URL: https://issues.apache.org/jira/browse/KAFKA-3086
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: Jakub Nowak
>  Labels: newbie
> Fix For: 0.9.1.0
>
>
> The following method is never used by MirrorMaker.
>   trait MirrorMakerMessageHandler {
> def handle(record: MessageAndMetadata[Array[Byte], Array[Byte]]): 
> util.List[ProducerRecord[Array[Byte], Array[Byte]]]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: MINOR: remove the init method from Serdes

[ymatsuda]

GitHub user ymatsuda opened a pull request:

https://github.com/apache/kafka/pull/834

MINOR: remove the init method from Serdes

@guozhangwang 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ymatsuda/kafka remove_init_from_Serdes

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/834.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #834


commit 8f432f47e2aaf2358152bee8330b53b7d8bf0b5a
Author: Yasuhiro Matsuda 
Date:   2016-01-28T00:32:16Z

MINOR: remove the init method from Serdes




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: KAFKA-3165: Fix ignored parameters in the grad...

[granthenke]

GitHub user granthenke opened a pull request:

https://github.com/apache/kafka/pull/835

KAFKA-3165: Fix ignored parameters in the gradle "All" tasks



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/granthenke/kafka all-parameters

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/835.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #835


commit d74fd508cba554f567e7ec62b9796ad1d5e93124
Author: Grant Henke 
Date:   2016-01-29T17:24:58Z

KAFKA-3165: Fix ignored parameters in the gradle "All" tasks




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3165) Fix ignored parameters in the gradle "All" tasks

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15123826#comment-15123826
 ] 

ASF GitHub Bot commented on KAFKA-3165:
---

GitHub user granthenke opened a pull request:

https://github.com/apache/kafka/pull/835

KAFKA-3165: Fix ignored parameters in the gradle "All" tasks



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/granthenke/kafka all-parameters

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/835.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #835


commit d74fd508cba554f567e7ec62b9796ad1d5e93124
Author: Grant Henke 
Date:   2016-01-29T17:24:58Z

KAFKA-3165: Fix ignored parameters in the gradle "All" tasks




> Fix ignored parameters in the gradle "All" tasks
> 
>
> Key: KAFKA-3165
> URL: https://issues.apache.org/jira/browse/KAFKA-3165
> Project: Kafka
>  Issue Type: Sub-task
>  Components: build
>Reporter: Grant Henke
>Assignee: Grant Henke
> Fix For: 0.9.1.0
>
>
> The "All" gradle tasks like "jarAll" or "testAll" do not respect passed 
> command line parameters. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk8 #328

[Apache Jenkins Server]

See 

Changes:

[wangguoz] KAFKA-3086: Remove unused method in MirrorMaker.

[wangguoz] MINOR: Fix spelling and grammar issues in ReplicaFetcherThread 
detailed

[wangguoz] KAFKA-3075; Fix ClassCastException in

--
[...truncated 82 lines...]
/x1/jenkins/jenkins-slave/workspace/kafka-trunk-jdk8/core/src/main/scala/kafka/server/KafkaServer.scala:305:
 a pure expression does nothing in statement position; you may be omitting 
necessary parentheses
ControllerStats.leaderElectionTimer
^
/x1/jenkins/jenkins-slave/workspace/kafka-trunk-jdk8/core/src/main/scala/kafka/controller/ControllerChannelManager.scala:389:
 class BrokerEndPoint in object UpdateMetadataRequest is deprecated: see 
corresponding Javadoc for more information.
  new UpdateMetadataRequest.BrokerEndPoint(brokerEndPoint.id, 
brokerEndPoint.host, brokerEndPoint.port)
^
/x1/jenkins/jenkins-slave/workspace/kafka-trunk-jdk8/core/src/main/scala/kafka/controller/ControllerChannelManager.scala:391:
 constructor UpdateMetadataRequest in class UpdateMetadataRequest is 
deprecated: see corresponding Javadoc for more information.
new UpdateMetadataRequest(controllerId, controllerEpoch, 
liveBrokers.asJava, partitionStates.asJava)
^
/x1/jenkins/jenkins-slave/workspace/kafka-trunk-jdk8/core/src/main/scala/kafka/network/BlockingChannel.scala:129:
 method readFromReadableChannel in class NetworkReceive is deprecated: see 
corresponding Javadoc for more information.
  response.readFromReadableChannel(channel)
   ^
there were 15 feature warning(s); re-run with -feature for details
11 warnings found
warning: [options] bootstrap class path not set in conjunction with -source 1.7
1 warning
:kafka-trunk-jdk8:core:processResources UP-TO-DATE
:kafka-trunk-jdk8:core:classes
:kafka-trunk-jdk8:clients:compileTestJavawarning: [options] bootstrap class 
path not set in conjunction with -source 1.7
Note: 
/x1/jenkins/jenkins-slave/workspace/kafka-trunk-jdk8/clients/src/test/java/org/apache/kafka/common/requests/RequestResponseTest.java
 uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
1 warning

:kafka-trunk-jdk8:clients:processTestResources
:kafka-trunk-jdk8:clients:testClasses
:kafka-trunk-jdk8:core:copyDependantLibs
:kafka-trunk-jdk8:core:copyDependantTestLibs
:kafka-trunk-jdk8:core:jar
:jar_core_2_11
Building project 'core' with Scala version 2.11.7
:kafka-trunk-jdk8:clients:compileJava UP-TO-DATE
:kafka-trunk-jdk8:clients:processResources UP-TO-DATE
:kafka-trunk-jdk8:clients:classes UP-TO-DATE
:kafka-trunk-jdk8:clients:determineCommitId UP-TO-DATE
:kafka-trunk-jdk8:clients:createVersionFile
:kafka-trunk-jdk8:clients:jar UP-TO-DATE
:kafka-trunk-jdk8:core:compileJava UP-TO-DATE
:kafka-trunk-jdk8:core:compileScalaJava HotSpot(TM) 64-Bit Server VM warning: 
ignoring option MaxPermSize=512m; support was removed in 8.0

/x1/jenkins/jenkins-slave/workspace/kafka-trunk-jdk8/core/src/main/scala/kafka/api/OffsetCommitRequest.scala:79:
 value DEFAULT_TIMESTAMP in object OffsetCommitRequest is deprecated: see 
corresponding Javadoc for more information.

org.apache.kafka.common.requests.OffsetCommitRequest.DEFAULT_TIMESTAMP
 ^
/x1/jenkins/jenkins-slave/workspace/kafka-trunk-jdk8/core/src/main/scala/kafka/common/OffsetMetadataAndError.scala:36:
 value DEFAULT_TIMESTAMP in object OffsetCommitRequest is deprecated: see 
corresponding Javadoc for more information.
 commitTimestamp: Long = 
org.apache.kafka.common.requests.OffsetCommitRequest.DEFAULT_TIMESTAMP,

  ^
/x1/jenkins/jenkins-slave/workspace/kafka-trunk-jdk8/core/src/main/scala/kafka/common/OffsetMetadataAndError.scala:37:
 value DEFAULT_TIMESTAMP in object OffsetCommitRequest is deprecated: see 
corresponding Javadoc for more information.
 expireTimestamp: Long = 
org.apache.kafka.common.requests.OffsetCommitRequest.DEFAULT_TIMESTAMP) {

  ^
/x1/jenkins/jenkins-slave/workspace/kafka-trunk-jdk8/core/src/main/scala/kafka/coordinator/GroupMetadataManager.scala:395:
 value DEFAULT_TIMESTAMP in object OffsetCommitRequest is deprecated: see 
corresponding Javadoc for more information.
  if (value.expireTimestamp == 
org.apache.kafka.common.requests.OffsetCommitRequest.DEFAULT_TIMESTAMP)

^
/x1/jenkins/jenkins-slave/workspace/kafka-trunk-jdk8/core/src/main/scala/kafka/server/KafkaApis.scala:293:
 value DEFAULT_TIMESTAMP 

[jira] [Commented] (KAFKA-3110) can't set cluster acl for a user to CREATE topics without first creating a topic

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15123851#comment-15123851
 ] 

Ismael Juma commented on KAFKA-3110:


Thanks for checking.

> can't set cluster acl for a user to CREATE topics without first creating a 
> topic
> 
>
> Key: KAFKA-3110
> URL: https://issues.apache.org/jira/browse/KAFKA-3110
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.9.0.0
>Reporter: Thomas Graves
>
> I started a new kafka cluster with security.  I tried to give a user cluster 
> CREATE permissions so they could create topics:
> kafka-acls.sh --authorizer-properties zookeeper.connect=host.com:2181 
> --cluster --add --operation CREATE --allow-principal User:myuser
> This failed with the error below and the broker ended up shutting down and 
> wouldn't restart without removing the zookeeper data.
> @40005699398806bd170c org.I0Itec.zkclient.exception.ZkNoNodeException: 
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = 
> NoNode for /kafka-acl/Topic
> To work around this you can first create any topic which creates the 
> zookeeper node and then after that you can give the user create permissions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2143) Replicas get ahead of leader and fail

[Jiangjie Qin (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15123866#comment-15123866
 ] 

Jiangjie Qin commented on KAFKA-2143:
-

Thanks, [~guozhang]. I almost forgot I had this ticket :)

> Replicas get ahead of leader and fail
> -
>
> Key: KAFKA-2143
> URL: https://issues.apache.org/jira/browse/KAFKA-2143
> Project: Kafka
>  Issue Type: Bug
>  Components: replication
>Affects Versions: 0.8.2.1
>Reporter: Evan Huus
>Assignee: Jiangjie Qin
> Fix For: 0.9.0.1
>
>
> On a cluster of 6 nodes, we recently saw a case where a single 
> under-replicated partition suddenly appeared, replication lag spiked, and 
> network IO spiked. The cluster appeared to recover eventually on its own,
> Looking at the logs, the thing which failed was partition 7 of the topic 
> {{background_queue}}. It had an ISR of 1,4,3 and its leader at the time was 
> 3. Here are the interesting log lines:
> On node 3 (the leader):
> {noformat}
> [2015-04-23 16:50:05,879] ERROR [Replica Manager on Broker 3]: Error when 
> processing fetch request for partition [background_queue,7] offset 3722949957 
> from follower with correlation id 148185816. Possible cause: Request for 
> offset 3722949957 but we only have log segments in the range 3648049863 to 
> 3722949955. (kafka.server.ReplicaManager)
> [2015-04-23 16:50:05,879] ERROR [Replica Manager on Broker 3]: Error when 
> processing fetch request for partition [background_queue,7] offset 3722949957 
> from follower with correlation id 156007054. Possible cause: Request for 
> offset 3722949957 but we only have log segments in the range 3648049863 to 
> 3722949955. (kafka.server.ReplicaManager)
> [2015-04-23 16:50:13,960] INFO Partition [background_queue,7] on broker 3: 
> Shrinking ISR for partition [background_queue,7] from 1,4,3 to 3 
> (kafka.cluster.Partition)
> {noformat}
> Note that both replicas suddenly asked for an offset *ahead* of the available 
> offsets.
> And on nodes 1 and 4 (the replicas) many occurrences of the following:
> {noformat}
> [2015-04-23 16:50:05,935] INFO Scheduling log segment 3648049863 for log 
> background_queue-7 for deletion. (kafka.log.Log) (edited)
> {noformat}
> Based on my reading, this looks like the replicas somehow got *ahead* of the 
> leader, asked for an invalid offset, got confused, and re-replicated the 
> entire topic from scratch to recover (this matches our network graphs, which 
> show 3 sending a bunch of data to 1 and 4).
> Taking a stab in the dark at the cause, there appears to be a race condition 
> where replicas can receive a new offset before the leader has committed it 
> and is ready to replicate?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3165) Fix ignored parameters in the gradle "All" tasks

[Grant Henke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Grant Henke updated KAFKA-3165:
---
Status: Patch Available  (was: Open)

> Fix ignored parameters in the gradle "All" tasks
> 
>
> Key: KAFKA-3165
> URL: https://issues.apache.org/jira/browse/KAFKA-3165
> Project: Kafka
>  Issue Type: Sub-task
>  Components: build
>Reporter: Grant Henke
>Assignee: Grant Henke
> Fix For: 0.9.1.0
>
>
> The "All" gradle tasks like "jarAll" or "testAll" do not respect passed 
> command line parameters. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: kafka-consumer-groups.sh doesn't work when consumers are off

[Cliff Rhyne]

Thanks for the added info.  For the mean time we'll rely on the older
ConsumerOffsetChecker for our monitoring tools.

Thanks,
Cliff

On Fri, Jan 29, 2016 at 10:56 AM, Guozhang Wang  wrote:

> Tao,
>
> You are right, ConsumerOffsetChecker can still get offsets from the offset
> manager in Kafka.
>
> Guozhang
>
> On Thu, Jan 28, 2016 at 9:36 PM, tao xiao  wrote:
>
> > it first issues an offsetrequest to broker and check if offset is stored
> in
> > broker if not it will queries zk
> >
> >
> >
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala#L171
> >
> > On Fri, 29 Jan 2016 at 13:11 Guozhang Wang  wrote:
> >
> > > Tao,
> > >
> > > Hmm that is a bit wired since ConsumerOffsetChecker itself does not
> talk
> > to
> > > brokers at all, but only through ZK.
> > >
> > > Guozhang
> > >
> > > On Thu, Jan 28, 2016 at 6:07 PM, tao xiao 
> wrote:
> > >
> > > > Guozhang,
> > > >
> > > > The old ConsumerOffsetChecker works for new consumer too with offset
> > > stored
> > > > in Kafka. I tested it with mirror maker with new consumer enabled. it
> > is
> > > > able to show offset during mirror maker running and after its
> shutdown.
> > > >
> > > > On Fri, 29 Jan 2016 at 06:34 Guozhang Wang 
> wrote:
> > > >
> > > > > Once the offset is written to the log it is persistent and hence
> > should
> > > > > survive broker failures. And its retention policy is configurable.
> > > > >
> > > > > It may be a bit misleading in saying "in-memory cache" in my
> previous
> > > > > email: the brokers just keep the in-memory map of [group,
> partition]
> > ->
> > > > > latest_offset, while the offset commit history is kept in the log.
> > When
> > > > we
> > > > > delete the group, we remove the corresponding entry from memory map
> > and
> > > > put
> > > > > a tombstone into log as well so that the old offsets will be
> > compacted
> > > > > eventually according to compaction policy.
> > > > >
> > > > > The old ConsumerOffsetChecker only works for old consumer that
> stores
> > > > > offset in ZK.
> > > > >
> > > > > Guozhang
> > > > >
> > > > > On Thu, Jan 28, 2016 at 1:43 PM, Cliff Rhyne 
> > wrote:
> > > > >
> > > > > > Hi Guozhang,
> > > > > >
> > > > > > That looks like it might help but feels like there might be some
> > > gaps.
> > > > > > Would it be able to survive restarts of the kafka broker?  How
> long
> > > > would
> > > > > > it stay in the cache (and is that configurable)?  If it expires
> > from
> > > > the
> > > > > > cache, what's the cache-miss operation look like?  (yes, a lot of
> > > this
> > > > > > depends on the data still being in the logs to recover)
> > > > > >
> > > > > > In the mean time, can I rely on the deprecated
> > ConsumerOffsetChecker
> > > > > (which
> > > > > > looks at zookeeper) even though I'm using the new KafkaConsumer?
> > > > > >
> > > > > > Thanks,
> > > > > > Cliff
> > > > > >
> > > > > > On Thu, Jan 28, 2016 at 3:30 PM, Guozhang Wang <
> wangg...@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Hi Cliff,
> > > > > > >
> > > > > > > Short answer to your question is it is just the current
> > > > implementation.
> > > > > > >
> > > > > > > The coordinator stores the offsets as messages in an internal
> > topic
> > > > and
> > > > > > > also keeps the latest offset values in in-memory. It answers
> > > > > > > ConsumerGroupRequest using its cached offset, and upon the
> > consumer
> > > > > group
> > > > > > > being removed since no member is alive already, it removed it
> > from
> > > > its
> > > > > > > in-memory cache and add a "tombstone" to the offset log as
> well.
> > > But
> > > > > the
> > > > > > > offsets are still persistent as messages in the log, which will
> > > only
> > > > be
> > > > > > > compacted after a while (this is depend on the log compaction
> > > > policy).
> > > > > > >
> > > > > > > There is a ticket open for improving on this scenario (
> > > > > > > https://issues.apache.org/jira/browse/KAFKA-2720) which lets
> the
> > > > > > > coordinator to only "purge" dead groups periodically instead of
> > > > > > > immediately, and that may partially resolve your case.
> > > > > > >
> > > > > > > Guozhang
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Jan 28, 2016 at 12:13 PM, Cliff Rhyne <
> crh...@signal.co>
> > > > > wrote:
> > > > > > >
> > > > > > > > Just following up on this concern.  Is there a constraint
> that
> > > > > prevents
> > > > > > > > ConsumerGroupCommand from reporting offsets on a group if no
> > > > members
> > > > > > are
> > > > > > > > connected, or is this just the current implementation?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Cliff
> > > > > > > >
> > > > > > > > On Mon, Jan 25, 2016 at 3:50 PM, Cliff Rhyne <
> crh...@signal.co
> > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > > I'm running into a few challenges trying to 

[jira] [Updated] (KAFKA-3075) java.util.HashMap cannot be cast to scala.collection.immutable.Map When using ZookeeperConsumerConnector.commitOffsets

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-3075:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> java.util.HashMap cannot be cast to scala.collection.immutable.Map When using 
>  ZookeeperConsumerConnector.commitOffsets
> ---
>
> Key: KAFKA-3075
> URL: https://issues.apache.org/jira/browse/KAFKA-3075
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.9.0.0
>Reporter: Pengwei
>Assignee: Ismael Juma
> Fix For: 0.9.0.1
>
>
> When using java api's commit offset :
>  public void commitOffsets(Map 
> offsetsToCommit, boolean retryOnFailure);
> and pass a  Java Hash Map to this interface, will found:
> java.lang.ClassCastException: java.util.HashMap cannot be cast to 
> scala.collection.immutable.Map
> at 
> kafka.javaapi.consumer.ZookeeperConsumerConnector.commitOffsets(ZookeeperConsumerConnector.scala:118)
> at 
> kafka.examples.CommitExceptionTest.testCommitNotExistTopicShoudThrowException(CommitExceptionTest.java:55)
> at kafka.examples.CommitExceptionTest.main(CommitExceptionTest.java:20)
> Test case testCommitNotExistTopicShoudThrowException OK.
> Exception in thread "main" java.lang.ClassCastException: java.util.HashMap 
> cannot be cast to scala.collection.immutable.Map
> at 
> kafka.javaapi.consumer.ZookeeperConsumerConnector.commitOffsets(ZookeeperConsumerConnector.scala:118)
> at 
> kafka.examples.CommitExceptionTest.testCommitOffsetOutOfRange(CommitExceptionTest.java:95)
> at kafka.examples.CommitExceptionTest.main(CommitExceptionTest.java:22)
> The Origin Code:
>  def commitOffsets(offsetsToCommit: java.util.Map[TopicAndPartition, 
> OffsetAndMetadata], retryOnFailure: Boolean) {
> 
> underlying.commitOffsets(offsetsToCommit.asInstanceOf[immutable.Map[TopicAndPartition,
>  OffsetAndMetadata]], retryOnFailure)
>   }
> I try to fix like this, it is OK:
>   def commitOffsets(offsetsToCommit: java.util.Map[TopicAndPartition, 
> OffsetAndMetadata], retryOnFailure: Boolean) {
> import scala.collection.JavaConverters._
> underlying.commitOffsets(offsetsToCommit.asScala.toMap, retryOnFailure)
>   }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3029) Make class org.apache.kafka.common.TopicPartition Serializable

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-3029:
-
   Resolution: Fixed
Fix Version/s: (was: 0.9.0.1)
   0.9.1.0
   Status: Resolved  (was: Patch Available)

Issue resolved by pull request 711
[https://github.com/apache/kafka/pull/711]

> Make class org.apache.kafka.common.TopicPartition Serializable
> --
>
> Key: KAFKA-3029
> URL: https://issues.apache.org/jira/browse/KAFKA-3029
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 0.9.0.0
>Reporter: Praveen Devarao
>  Labels: easyfix, patch
> Fix For: 0.9.1.0
>
>
> The client class TopicPartition is exposed and used by consumer applications 
> directly. In case where the application needs to checkpoint the state it is 
> difficult to serialize different app classes that use TopicPartition as 
> TopicParitition is not serializable.
> For instance consider the Spark use case where RDDs have to be 
> checkpointedthe KafakaInputDstream (which we are currently modifying to 
> use the new Kafka API rather than the Highlevel apis in previous version) 
> cannot be serialized due to above limitation.
> I have created a patch to serialize TopicPartition class by making it 
> implement serializable interface and have issued a pull request.
> Can this be merged for the next release (0.9.0.1)
> Thanks
> Praveen



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3029) Make class org.apache.kafka.common.TopicPartition Serializable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15123821#comment-15123821
 ] 

ASF GitHub Bot commented on KAFKA-3029:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/711


> Make class org.apache.kafka.common.TopicPartition Serializable
> --
>
> Key: KAFKA-3029
> URL: https://issues.apache.org/jira/browse/KAFKA-3029
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 0.9.0.0
>Reporter: Praveen Devarao
>  Labels: easyfix, patch
> Fix For: 0.9.1.0
>
>
> The client class TopicPartition is exposed and used by consumer applications 
> directly. In case where the application needs to checkpoint the state it is 
> difficult to serialize different app classes that use TopicPartition as 
> TopicParitition is not serializable.
> For instance consider the Spark use case where RDDs have to be 
> checkpointedthe KafakaInputDstream (which we are currently modifying to 
> use the new Kafka API rather than the Highlevel apis in previous version) 
> cannot be serialized due to above limitation.
> I have created a patch to serialize TopicPartition class by making it 
> implement serializable interface and have issued a pull request.
> Can this be merged for the next release (0.9.0.1)
> Thanks
> Praveen



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3029:Marking class org.apache.kafka.comm...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/711


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: KAFKA-2676: Fix incorrect package name in test...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/828


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2676) Incorrect package name in tests (unit and integration prefix)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15123835#comment-15123835
 ] 

ASF GitHub Bot commented on KAFKA-2676:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/828


> Incorrect package name in tests (unit and integration prefix)
> -
>
> Key: KAFKA-2676
> URL: https://issues.apache.org/jira/browse/KAFKA-2676
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: chen zhu
>  Labels: newbie
>
> AclCommandTest and a few others have the package unit.* or integration.*. We 
> should remove the unit or integration prefix.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-2676) Incorrect package name in tests (unit and integration prefix)

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-2676.
--
   Resolution: Fixed
Fix Version/s: 0.9.0.1

Issue resolved by pull request 828
[https://github.com/apache/kafka/pull/828]

> Incorrect package name in tests (unit and integration prefix)
> -
>
> Key: KAFKA-2676
> URL: https://issues.apache.org/jira/browse/KAFKA-2676
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: chen zhu
>  Labels: newbie
> Fix For: 0.9.0.1
>
>
> AclCommandTest and a few others have the package unit.* or integration.*. We 
> should remove the unit or integration prefix.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: MINOR: remove the init method from Serdes

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/834


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (KAFKA-3110) can't set cluster acl for a user to CREATE topics without first creating a topic

[Thomas Graves (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Graves resolved KAFKA-3110.
--
Resolution: Invalid

> can't set cluster acl for a user to CREATE topics without first creating a 
> topic
> 
>
> Key: KAFKA-3110
> URL: https://issues.apache.org/jira/browse/KAFKA-3110
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.9.0.0
>Reporter: Thomas Graves
>
> I started a new kafka cluster with security.  I tried to give a user cluster 
> CREATE permissions so they could create topics:
> kafka-acls.sh --authorizer-properties zookeeper.connect=host.com:2181 
> --cluster --add --operation CREATE --allow-principal User:myuser
> This failed with the error below and the broker ended up shutting down and 
> wouldn't restart without removing the zookeeper data.
> @40005699398806bd170c org.I0Itec.zkclient.exception.ZkNoNodeException: 
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = 
> NoNode for /kafka-acl/Topic
> To work around this you can first create any topic which creates the 
> zookeeper node and then after that you can give the user create permissions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3110) can't set cluster acl for a user to CREATE topics without first creating a topic

[Thomas Graves (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15123845#comment-15123845
 ] 

Thomas Graves commented on KAFKA-3110:
--

Tried to reproduce this again but couldn't.  I must have done something wrong 
initially.

> can't set cluster acl for a user to CREATE topics without first creating a 
> topic
> 
>
> Key: KAFKA-3110
> URL: https://issues.apache.org/jira/browse/KAFKA-3110
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.9.0.0
>Reporter: Thomas Graves
>
> I started a new kafka cluster with security.  I tried to give a user cluster 
> CREATE permissions so they could create topics:
> kafka-acls.sh --authorizer-properties zookeeper.connect=host.com:2181 
> --cluster --add --operation CREATE --allow-principal User:myuser
> This failed with the error below and the broker ended up shutting down and 
> wouldn't restart without removing the zookeeper data.
> @40005699398806bd170c org.I0Itec.zkclient.exception.ZkNoNodeException: 
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = 
> NoNode for /kafka-acl/Topic
> To work around this you can first create any topic which creates the 
> zookeeper node and then after that you can give the user create permissions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3029) Make class org.apache.kafka.common.TopicPartition Serializable

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-3029:
-
Fix Version/s: 0.9.0.1

> Make class org.apache.kafka.common.TopicPartition Serializable
> --
>
> Key: KAFKA-3029
> URL: https://issues.apache.org/jira/browse/KAFKA-3029
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 0.9.0.0
>Reporter: Praveen Devarao
>  Labels: easyfix, patch
> Fix For: 0.9.0.1, 0.9.1.0
>
>
> The client class TopicPartition is exposed and used by consumer applications 
> directly. In case where the application needs to checkpoint the state it is 
> difficult to serialize different app classes that use TopicPartition as 
> TopicParitition is not serializable.
> For instance consider the Spark use case where RDDs have to be 
> checkpointedthe KafakaInputDstream (which we are currently modifying to 
> use the new Kafka API rather than the Highlevel apis in previous version) 
> cannot be serialized due to above limitation.
> I have created a patch to serialize TopicPartition class by making it 
> implement serializable interface and have issued a pull request.
> Can this be merged for the next release (0.9.0.1)
> Thanks
> Praveen



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Confluence Access

[Bill Warshaw]

Hello,

I'm planning to create a new KIP, but I don't think I have the proper
permissions to actually create a child page on the Confluence wiki.  Could
an administrator give me access to create pages, or am I missing something
obvious?

Thanks,
Bill Warshaw

-- 
 
This message and any attachments are solely for the intended recipient. If 
you are not the intended recipient, disclosure, copying, use, or 
distribution of the information included in this message is prohibited -- 
please immediately and permanently delete this message.

[jira] [Commented] (KAFKA-3164) Document client and mirrormaker upgrade procedure/requirements

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15124312#comment-15124312
 ] 

ASF GitHub Bot commented on KAFKA-3164:
---

GitHub user granthenke opened a pull request:

https://github.com/apache/kafka/pull/838

KAFKA-3164: Document client and mirrormaker upgrade procedure/require…

…ments

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/granthenke/kafka upgrade-docs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/838.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #838


commit e855bfb711348b0fd880d7f4fc2c54cd31e80ef8
Author: Grant Henke 
Date:   2016-01-29T22:09:04Z

KAFKA-3164: Document client and mirrormaker upgrade procedure/requirements




> Document client and mirrormaker upgrade procedure/requirements
> --
>
> Key: KAFKA-3164
> URL: https://issues.apache.org/jira/browse/KAFKA-3164
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.9.0.0
>Reporter: Grant Henke
>Assignee: Grant Henke
>Priority: Minor
>
> Many users in the mailing list have asked questions about new clients working 
> on old brokers, and mirrormaker breaking when upgrading to 0.9.0. Adding a 
> section to the upgrade docs to let users know to upgrade brokers before 
> clients and downstream mirrormaker first should help other users from making 
> the same mistake. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Contributor List

[Abraham Fine]

Would it be possible to add me to the contributor list? My apache jira username 
is: abrahamfine

Thanks,
Abe

[jira] [Commented] (KAFKA-3121) KStream DSL API Improvement

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15124460#comment-15124460
 ] 

ASF GitHub Bot commented on KAFKA-3121:
---

GitHub user guozhangwang opened a pull request:

https://github.com/apache/kafka/pull/839

KAFKA-3121: Refactor KStream Aggregate to be Lambda-able.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/guozhangwang/kafka K3121s2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/839.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #839


commit bf4c4cb3dbb5b4066d9c3e0ada5b7ffd98eb129a
Author: Guozhang Wang 
Date:   2016-01-14T20:27:58Z

add internal source topic for tracking

commit 1485dff08a76c6ff685b0fe72226ce3b629b1d3c
Author: Guozhang Wang 
Date:   2016-01-14T22:32:08Z

minor fix for this.interSourceTopics

commit 60cafd0885c41f93e408f8d89880187ddec789a1
Author: Guozhang Wang 
Date:   2016-01-15T01:09:00Z

add KStream windowed aggregation

commit 983a626008d987828deabe45d75e26e909032843
Author: Guozhang Wang 
Date:   2016-01-15T01:34:56Z

merge from apache trunk

commit 57051720de4238feb4dc3c505053096042a87d9c
Author: Guozhang Wang 
Date:   2016-01-15T21:38:53Z

v1

commit 4a49205fcab3a05ed1fd05a34c7a9a92794b992d
Author: Guozhang Wang 
Date:   2016-01-15T22:07:17Z

minor fix on HoppingWindows

commit 9b4127e91c3a551fb655155d9b8e0df50132d0b7
Author: Guozhang Wang 
Date:   2016-01-15T22:43:14Z

fix HoppingWindows

commit 9649fe5c8a9b2e900e7746ae7b8745bb65694583
Author: Guozhang Wang 
Date:   2016-01-16T19:00:54Z

add retainDuplicate option in RocksDBWindowStore

commit 8a9ea02ac3f9962416defa79d16069431063eac0
Author: Guozhang Wang 
Date:   2016-01-16T19:06:12Z

minor fixes

commit 4123528cf4695b05235789ebfca3a63e8a832ffa
Author: Guozhang Wang 
Date:   2016-01-18T17:55:02Z

Merge branch 'trunk' of https://git-wip-us.apache.org/repos/asf/kafka into 
K3104

commit 46e8c8d285c0afae6da9ec7437d082060599f3f1
Author: Guozhang Wang 
Date:   2016-01-18T19:15:47Z

add wordcount and pipe jobs

commit 582d3ac24bfe08edb1c567461971cd35c1f75a00
Author: Guozhang Wang 
Date:   2016-01-18T21:53:21Z

merge from trunk

commit 5a002fadfcf760627274ddaa016deeaed5a3199f
Author: Guozhang Wang 
Date:   2016-01-19T00:06:34Z

1. WallClockTimestampExtractor as default; 2. remove windowMS config; 3. 
override state dir with jobId prefix;

commit 7425673e523c42806b29a364564a747443712a53
Author: Guozhang Wang 
Date:   2016-01-19T01:26:11Z

Add PageViewJob

commit ca04ba8d18674c521ad67872562a7671cb0e2c0d
Author: Guozhang Wang 
Date:   2016-01-19T06:23:05Z

minor changes on topic names

commit 563cc546b3a0dd16d586d2df33c37d2c5a5bfb18
Author: Guozhang Wang 
Date:   2016-01-19T21:30:11Z

change config importance levels

commit 4218904505363e61bb4c6b60dc5b13badfd39697
Author: Guozhang Wang 
Date:   2016-01-21T00:11:34Z

Merge branch 'trunk' of https://git-wip-us.apache.org/repos/asf/kafka into 
K3066

commit 26fb5f3f5a8c9b304c5b1e61778c6bc1d9d5fccb
Author: Guozhang Wang 
Date:   2016-01-21T06:43:04Z

demo examples v1

commit 6d92a55d770e058183daabb7aaef7675335fbbad
Author: Guozhang Wang 
Date:   2016-01-22T00:41:12Z

Merge branch 'trunk' of https://git-wip-us.apache.org/repos/asf/kafka into 
K3066

commit 929e405058eb61d38510120f3f3ed50cd0cfaf47
Author: Guozhang Wang 
Date:   2016-01-22T01:02:04Z

add RollingRocksDBStore

commit 324eb584b97ed3c228347d108d697d2f5133ea99
Author: Guozhang Wang 
Date:   2016-01-22T01:23:32Z

modify MeteredWindowStore

commit 7ba2d90fe1de1ca776cea23ff1c2e8f8b3a6c3f2
Author: Guozhang Wang 
Date:   2016-01-22T01:35:10Z

remove getter

commit a4d78bac9d84dfd1c7dab4ae465b9115ddc451b3
Author: Guozhang Wang 
Date:   2016-01-22T01:36:51Z

remove RollingRocksDB

commit d0e8198ac6a25315d7ab8d21894acf0077f88fde
Author: Guozhang Wang 
Date:   2016-01-22T17:24:32Z

adding cache layer on RocksDB

commit 257b53d3b6df967f8a015a06c9e178d4219d0f8c
Author: Guozhang Wang 
Date:   2016-01-22T23:15:08Z

dummy

commit 25fd73107c577ac2e4b32300d4fe132ad7ff7312
Author: Guozhang Wang 
Date:   2016-01-22T23:21:29Z

merge from trunk

commit 

[jira] [Created] (KAFKA-3173) Error while moving some partitions to OnlinePartition state

[Flavio Junqueira (JIRA)]

Flavio Junqueira created KAFKA-3173:
---

 Summary: Error while moving some partitions to OnlinePartition 
state 
 Key: KAFKA-3173
 URL: https://issues.apache.org/jira/browse/KAFKA-3173
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.9.0.0
Reporter: Flavio Junqueira
Priority: Critical
 Fix For: 0.9.0.1


We observed another instance of the problem reported in KAFKA-2300, but this 
time the error appeared in the partition state machine. In KAFKA-2300, we 
haven't cleaned up the state in {{PartitionStateMachine}} and 
{{ReplicaStateMachine}} as we do in {{KafkaController}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-42: Add Producer and Consumer Interceptors

[Jay Kreps]

The rationale for the CRC covering the whole record was to check corruption
in the full record contents as corruption there will equally prevent
someone trying to consume the data. I think you could argue either way but
let's definitely not end up with two different CRC calculations, that would
just be indecisive.

That is a super interesting finding about the CRC class. I think you are
right that it became an intrinsic in java 8 with hw support if available.
We should really switch to that. The CRC calculation is still one of the
biggest bottlenecks in the producer and consumer so that should
significantly speed things up.

-Jay

On Fri, Jan 29, 2016 at 4:30 PM, Becket Qin  wrote:

> Hi Anna,
>
> I think there is value if CRC for only user bytes can be used. This will
> help when we have future protocol updates. Otherwise any protocol migration
> might break auditing if it largely relies on CRC including system bytes.
>
> I did some test to understand the performance overhead of having a separate
> user bytes CRC,
>
> On my desktop, it took ~4.7 seconds to compute CRC for 10G bytes. So for
> each megabyte it takes 0.47 ms.
> To compare with compression cost, I used the compressor used by producer.
> The test uses GZIP and compress 1MB of data for each batch.
> It takes:
> ~90 seconds to compress 10GB bytes that contains all same byte.
> ~470 seconds to compress 10GB bytes containing random bytes.
> >1000 seconds to compress 10GB bytes with 4, 8, 16, 32 and 64 patterns. (I
> guess it takes time to update the pattern mapping)
>
> So the overhead of a separate CRC computing is <1% considering the producer
> do a lot of things other than compression.
>
> I am not sure if this overhead is worth taking or not, because we do have a
> huge number of messages to send, so any overhead should be avoided if
> possible.
>
> BTW. Another interesting finding is that the Crc32 class used in clients
> which used to be faster than java CRC32 in Java 1.6. It seems no longer the
> case. What I see is that Java CRC32 class is 2x faster than the Crc32 class
> we are using now.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
>
> On Fri, Jan 29, 2016 at 1:46 PM, Anna Povzner  wrote:
>
> > Joel, thanks for your feedback. I updated the wiki based on your comments
> > about the wiki writeup.
> >
> >
> > On Fri, Jan 29, 2016 at 11:50 AM, Anna Povzner 
> wrote:
> >
> > > Becket,
> > >
> > > In your scenario with one message from producer A and one message from
> > > producer B, those are two different messages, and they should be
> tracked
> > as
> > > two different messages. So I would argue for using record CRC -- CRC
> that
> > > is actually used by the system + it will not require computing a
> > different
> > > CRC again which will add performance overhead.
> > >
> > > If the broker ever changes the CRC, the scenarios when that happens
> > should
> > > be very well defined. As far as I know, the scenarios when CRC is
> > > overwritten by the broker (including KIP-31/32 changes):
> > > -- topic config is LogAppendTime for timestamp type
> > > -- upgrade/downgrade
> > > -- compression codec change (which could be inferred from config).
> > >
> > > Monitoring/audit just needs to know when CRCs are safe to use, which is
> > > most often is known from config. In the future, this can be further
> > > addressed by broker interceptors.
> > >
> > > Thanks,
> > > Anna
> > >
> > >
> > >
> > >
> > > On Fri, Jan 29, 2016 at 11:30 AM, Becket Qin 
> > wrote:
> > >
> > >> Neha,
> > >>
> > >> CRC is definitely an important type of metadata of a record. I am not
> > >> arguing about that. But I think we should distinguish between two
> types
> > of
> > >> checksum here, 1) the checksum of user data. and 2) the checksum
> > including
> > >> system appended bytes.
> > >>
> > >> I completely agree that (1) is good to add. But I am not sure if we
> > should
> > >> expose (2) to user, because this means any underlying protocol change
> > will
> > >> give a different CRC for exact same message. For example, let's say
> > >> producer A is sending message with timestamp. Producer B is sending
> > >> message
> > >> without timestamp. Even they are given the exact same message, the CRC
> > >> returned would be different.
> > >>
> > >> Also, Kafka broker will modify the system appended bytes in different
> > >> scenarios, such as compression codec change, message format
> > >> conversion(After KIP-31 and KIP-32).
> > >>
> > >> So my concern is that we are exposing CRC which including system
> > appended
> > >> bytes to user.
> > >>
> > >> Other than this I think everything looks good. Nice work, Anna.
> > >>
> > >> Thanks,
> > >>
> > >> Jiangjie (Becket) Qin
> > >>
> > >> On Fri, Jan 29, 2016 at 8:11 AM, Joel Koshy 
> > wrote:
> > >>
> > >> > Responding to some of the earlier comments in the thread:
> > >> >
> > >> > @Jay/@Neha,
> > >> >
> > >> > I think any one 

Re: [DISCUSS] KIP-42: Add Producer and Consumer Interceptors

[Becket Qin]

Hi Anna,

I think there is value if CRC for only user bytes can be used. This will
help when we have future protocol updates. Otherwise any protocol migration
might break auditing if it largely relies on CRC including system bytes.

I did some test to understand the performance overhead of having a separate
user bytes CRC,

On my desktop, it took ~4.7 seconds to compute CRC for 10G bytes. So for
each megabyte it takes 0.47 ms.
To compare with compression cost, I used the compressor used by producer.
The test uses GZIP and compress 1MB of data for each batch.
It takes:
~90 seconds to compress 10GB bytes that contains all same byte.
~470 seconds to compress 10GB bytes containing random bytes.
>1000 seconds to compress 10GB bytes with 4, 8, 16, 32 and 64 patterns. (I
guess it takes time to update the pattern mapping)

So the overhead of a separate CRC computing is <1% considering the producer
do a lot of things other than compression.

I am not sure if this overhead is worth taking or not, because we do have a
huge number of messages to send, so any overhead should be avoided if
possible.

BTW. Another interesting finding is that the Crc32 class used in clients
which used to be faster than java CRC32 in Java 1.6. It seems no longer the
case. What I see is that Java CRC32 class is 2x faster than the Crc32 class
we are using now.

Thanks,

Jiangjie (Becket) Qin


On Fri, Jan 29, 2016 at 1:46 PM, Anna Povzner  wrote:

> Joel, thanks for your feedback. I updated the wiki based on your comments
> about the wiki writeup.
>
>
> On Fri, Jan 29, 2016 at 11:50 AM, Anna Povzner  wrote:
>
> > Becket,
> >
> > In your scenario with one message from producer A and one message from
> > producer B, those are two different messages, and they should be tracked
> as
> > two different messages. So I would argue for using record CRC -- CRC that
> > is actually used by the system + it will not require computing a
> different
> > CRC again which will add performance overhead.
> >
> > If the broker ever changes the CRC, the scenarios when that happens
> should
> > be very well defined. As far as I know, the scenarios when CRC is
> > overwritten by the broker (including KIP-31/32 changes):
> > -- topic config is LogAppendTime for timestamp type
> > -- upgrade/downgrade
> > -- compression codec change (which could be inferred from config).
> >
> > Monitoring/audit just needs to know when CRCs are safe to use, which is
> > most often is known from config. In the future, this can be further
> > addressed by broker interceptors.
> >
> > Thanks,
> > Anna
> >
> >
> >
> >
> > On Fri, Jan 29, 2016 at 11:30 AM, Becket Qin 
> wrote:
> >
> >> Neha,
> >>
> >> CRC is definitely an important type of metadata of a record. I am not
> >> arguing about that. But I think we should distinguish between two types
> of
> >> checksum here, 1) the checksum of user data. and 2) the checksum
> including
> >> system appended bytes.
> >>
> >> I completely agree that (1) is good to add. But I am not sure if we
> should
> >> expose (2) to user, because this means any underlying protocol change
> will
> >> give a different CRC for exact same message. For example, let's say
> >> producer A is sending message with timestamp. Producer B is sending
> >> message
> >> without timestamp. Even they are given the exact same message, the CRC
> >> returned would be different.
> >>
> >> Also, Kafka broker will modify the system appended bytes in different
> >> scenarios, such as compression codec change, message format
> >> conversion(After KIP-31 and KIP-32).
> >>
> >> So my concern is that we are exposing CRC which including system
> appended
> >> bytes to user.
> >>
> >> Other than this I think everything looks good. Nice work, Anna.
> >>
> >> Thanks,
> >>
> >> Jiangjie (Becket) Qin
> >>
> >> On Fri, Jan 29, 2016 at 8:11 AM, Joel Koshy 
> wrote:
> >>
> >> > Responding to some of the earlier comments in the thread:
> >> >
> >> > @Jay/@Neha,
> >> >
> >> > I think any one of onCommit/onAppend/onArrival would work for the
> >> concrete
> >> > use-case that I had outlined. I think onArrival is additionally useful
> >> for
> >> > custom validation - i.e., reject the message and do not append if it
> >> > violates some cluster-specific rule (for e.g., if some header
> timestamp
> >> is
> >> > older than xyz). However, the thing with user-supplied validation is
> we
> >> > would have to do with a (new) generic error code in the producer
> >> response.
> >> > While there is a risk of a broker interceptor having high latency I
> >> think
> >> > that is acceptable since it is the user's responsibility to ensure low
> >> > latency - the producer call-back and onAcknowledgment interceptor are
> >> > similar in this regard although those are less risky. Even so, I think
> >> > there are clear use-cases for broker interceptors so I feel the risk
> >> part
> >> > is something that just needs to be documented. 

[jira] [Resolved] (KAFKA-3171) min.insync.replicas not respected

[Victor Serbu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Victor Serbu resolved KAFKA-3171.
-
Resolution: Invalid

You are right, after I changed ack with "acks" producer raise error when ISR 
are less than minimum configured.

{code}
1454106693 Send message: (0, {"id":0,"msg_text":"aa"})
java.util.concurrent.ExecutionException: 
org.apache.kafka.common.errors.NotEnoughReplicasException: Messages are 
rejected since there are fewer in-sync replicas than required.
at 
org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:56)
at 
org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:43)
at 
org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:25)
at kafka.examples.Producer.run(Producer.java:73)
Caused by: org.apache.kafka.common.errors.NotEnoughReplicasException: Messages 
are rejected since there are fewer in-sync replicas than required.
{code}

I haven't seen any error on stdout/stderr when I used a wrong parameter. I will 
try to configure log4j for my test program.

Sorry that I didn't verified this, the documentation also specify correct 
parameter "acks".

> min.insync.replicas not respected
> -
>
> Key: KAFKA-3171
> URL: https://issues.apache.org/jira/browse/KAFKA-3171
> Project: Kafka
>  Issue Type: Bug
> Environment: Kafka downloaded from 
> http://apache.javapipe.com/kafka/0.9.0.0/kafka_2.11-0.9.0.0.tgz
>Reporter: Victor Serbu
>Assignee: Grant Henke
>
> Hello.
> I wanted to test kafka durability properties but I didn't managed to block 
> producer if number of ISR is less than min.insync.replicas.
> For my test I used 3 VM. On each VM is running kafka and zookeeper.
> Then I created a topic with min ISR = 2.
> {code}
> [root@kafka1 kafka_2.11-0.9.0.0]# bin/kafka-topics.sh --create --zookeeper 
> kafka1.novalocal:2181,kafka2.novalocal:2181,kafka3.novalocal:2181 
> --replication-factor 3 --partitions 1 --config 'min.insync.replicas=3' 
> --topic test61
> Created topic "test61".
> [root@kafka1 kafka_2.11-0.9.0.0]# bin/kafka-topics.sh --describe --zookeeper 
> kafka1.novalocal:2181,kafka2.novalocal:2181,kafka3.novalocal:2181 --topic 
> test60  Topic:test60
> PartitionCount:1ReplicationFactor:3 Configs:min.insync.replicas=2
> Topic: test60   Partition: 0Leader: 0   Replicas: 0,2,1 Isr: 
> 0,2,1
> {code}
> Then I stopped two of kafka brokers and started a producers and all messages 
> were commited (using a consumer I saw all the messages).
> {code}
> [root@kafka1 kafka_2.11-0.9.0.0]# bin/kafka-topics.sh --describe --zookeeper 
> kafka1.novalocal:2181,kafka2.novalocal:2181,kafka3.novalocal:2181 --topic 
> test60
> Topic:test60PartitionCount:1ReplicationFactor:3 
> Configs:min.insync.replicas=2
> Topic: test60   Partition: 0Leader: 0   Replicas: 0,2,1 Isr: 0
> {code}
> Producer
> {code}
> /**
>  * Licensed to the Apache Software Foundation (ASF) under one or more
>  * contributor license agreements.  See the NOTICE file distributed with
>  * this work for additional information regarding copyright ownership.
>  * The ASF licenses this file to You under the Apache License, Version 2.0
>  * (the "License"); you may not use this file except in compliance with
>  * the License.  You may obtain a copy of the License at
>  *
>  *http://www.apache.org/licenses/LICENSE-2.0
>  *
>  * Unless required by applicable law or agreed to in writing, software
>  * distributed under the License is distributed on an "AS IS" BASIS,
>  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>  * See the License for the specific language governing permissions and
>  * limitations under the License.
>  */
> package kafka.examples;
> import java.util.Properties;
> import java.util.Map;
> import java.util.Iterator;
> import java.util.concurrent.ExecutionException;
> import org.apache.kafka.clients.producer.Callback;
> import org.apache.kafka.clients.producer.KafkaProducer;
> import org.apache.kafka.clients.producer.ProducerRecord;
> import org.apache.kafka.clients.producer.RecordMetadata;
> import org.apache.kafka.common.Metric;
> import org.apache.kafka.common.MetricName;
> public class Producer extends Thread
> {
>   private final KafkaProducer producer;
>   private final String topic;
>   private final Boolean isAsync;
>   private final Integer maxId;
>   private final Boolean debug;
>   public Producer(String topic, Boolean isAsync, Integer maxId, Boolean debug)
>   {
> Properties props = new Properties();
> props.put("bootstrap.servers", 
> 

Build failed in Jenkins: kafka-trunk-jdk7 #1003

[Apache Jenkins Server]

See 

Changes:

[wangguoz] MINOR: Remove unused imports from tests whose packages were fixed

--
[...truncated 2793 lines...]

kafka.log.LogTest > testTimeBasedLogRoll PASSED

kafka.log.LogTest > testLoadEmptyLog PASSED

kafka.log.LogTest > testMessageSetSizeCheck PASSED

kafka.log.LogTest > testIndexResizingAtTruncation PASSED

kafka.log.LogTest > testCompactedTopicConstraints PASSED

kafka.log.LogTest > testThatGarbageCollectingSegmentsDoesntChangeOffset PASSED

kafka.log.LogTest > testAppendAndReadWithSequentialOffsets PASSED

kafka.log.LogTest > testParseTopicPartitionNameForNull PASSED

kafka.log.LogTest > testAppendAndReadWithNonSequentialOffsets PASSED

kafka.log.LogTest > testParseTopicPartitionNameForMissingSeparator PASSED

kafka.log.LogTest > testCorruptIndexRebuild PASSED

kafka.log.LogTest > testBogusIndexSegmentsAreRemoved PASSED

kafka.log.LogTest > testCompressedMessages PASSED

kafka.log.LogTest > testAppendMessageWithNullPayload PASSED

kafka.log.LogTest > testCorruptLog PASSED

kafka.log.LogTest > testLogRecoversToCorrectOffset PASSED

kafka.log.LogTest > testReopenThenTruncate PASSED

kafka.log.LogTest > testParseTopicPartitionNameForMissingPartition PASSED

kafka.log.LogTest > testParseTopicPartitionNameForEmptyName PASSED

kafka.log.LogTest > testOpenDeletesObsoleteFiles PASSED

kafka.log.LogTest > testSizeBasedLogRoll PASSED

kafka.log.LogTest > testTimeBasedLogRollJitter PASSED

kafka.log.LogTest > testParseTopicPartitionName PASSED

kafka.log.LogTest > testTruncateTo PASSED

kafka.log.LogTest > testCleanShutdownFile PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[0] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[1] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[2] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[3] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[4] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[5] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[6] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[7] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[8] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[9] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[10] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[11] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[12] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[13] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[14] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[15] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[16] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[17] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[18] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[19] PASSED

kafka.producer.SyncProducerTest > testReachableServer PASSED

kafka.producer.SyncProducerTest > testMessageSizeTooLarge PASSED

kafka.producer.SyncProducerTest > testNotEnoughReplicas PASSED

kafka.producer.SyncProducerTest > testMessageSizeTooLargeWithAckZero PASSED

kafka.producer.SyncProducerTest > testProducerCanTimeout PASSED

kafka.producer.SyncProducerTest > testProduceRequestWithNoResponse PASSED

kafka.producer.SyncProducerTest > testEmptyProduceRequest PASSED

kafka.producer.SyncProducerTest > testProduceCorrectlyReceivesResponse PASSED

kafka.producer.AsyncProducerTest > testFailedSendRetryLogic PASSED

kafka.producer.AsyncProducerTest > testQueueTimeExpired PASSED

kafka.producer.AsyncProducerTest > testPartitionAndCollateEvents PASSED

kafka.producer.AsyncProducerTest > testBatchSize PASSED

kafka.producer.AsyncProducerTest > testSerializeEvents PASSED

kafka.producer.AsyncProducerTest > testProducerQueueSize PASSED

kafka.producer.AsyncProducerTest > testRandomPartitioner PASSED

kafka.producer.AsyncProducerTest > testInvalidConfiguration PASSED

kafka.producer.AsyncProducerTest > testInvalidPartition PASSED

kafka.producer.AsyncProducerTest > testNoBroker PASSED

kafka.producer.AsyncProducerTest > testProduceAfterClosed PASSED

kafka.producer.AsyncProducerTest > testJavaProducer PASSED

kafka.producer.AsyncProducerTest > testIncompatibleEncoder PASSED

kafka.producer.ProducerTest > testSendToNewTopic PASSED

kafka.producer.ProducerTest > testAsyncSendCanCorrectlyFailWithTimeout PASSED

kafka.producer.ProducerTest > testSendNullMessage PASSED

kafka.producer.ProducerTest > testUpdateBrokerPartitionInfo PASSED

kafka.producer.ProducerTest > testSendWithDeadBroker PASSED

kafka.tools.ConsoleConsumerTest > shouldLimitReadsToMaxMessageLimit PASSED

kafka.tools.ConsoleConsumerTest > shouldParseValidNewConsumerValidConfig PASSED

[jira] [Created] (KAFKA-3174) Re-evaluate the CRC32 class performance.

[Jiangjie Qin (JIRA)]

Jiangjie Qin created KAFKA-3174:
---

 Summary: Re-evaluate the CRC32 class performance.
 Key: KAFKA-3174
 URL: https://issues.apache.org/jira/browse/KAFKA-3174
 Project: Kafka
  Issue Type: Improvement
Reporter: Jiangjie Qin


We org.apache.kafka.common.utils.CRC32 because it has better performance than 
java.util.zip.CRC32 in Java 1.6.

In a recent test I ran it looks in Java 1.8 the CRC32 class is 2x as fast as 
the Crc32 class we are using now. We may want to re-evaluate the performance of 
Crc32 class and see it makes sense to simply use java CRC32 instead.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3164) Document client and mirrormaker upgrade procedure/requirements

[Grant Henke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Grant Henke updated KAFKA-3164:
---
Status: Patch Available  (was: Open)

> Document client and mirrormaker upgrade procedure/requirements
> --
>
> Key: KAFKA-3164
> URL: https://issues.apache.org/jira/browse/KAFKA-3164
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.9.0.0
>Reporter: Grant Henke
>Assignee: Grant Henke
>Priority: Minor
>
> Many users in the mailing list have asked questions about new clients working 
> on old brokers, and mirrormaker breaking when upgrading to 0.9.0. Adding a 
> section to the upgrade docs to let users know to upgrade brokers before 
> clients and downstream mirrormaker first should help other users from making 
> the same mistake. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)