date:20160331

Re: KIP-4 Wiki Update

2016-03-31 Thread Grant Henke

I have update the wiki and patch based on the feedback on the metadata
changes. Please take a look and let me know if there are any concerns or
issues.


   - Wiki:
   
https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-MetadataSchema
   - PR: https://github.com/apache/kafka/pull/1095

I will hold a vote on the metadata change soon if no major issues are
raised.

On Thu, Mar 31, 2016 at 6:33 PM, Jason Gustafson  wrote:

> I also prefer B. In addition to being intuitive, it seems less error-prone
> in the long term, though it might be a little annoying for clients
> maintaining backwards compatibility.
>
> -Jason
>
> On Thu, Mar 31, 2016 at 1:24 PM, Ismael Juma  wrote:
>
> > I prefer B, the fact that we version the protocol means that we can fix
> > mistakes instead of living with them forever. We should take advantage of
> > that.
> >
> > Ismael
> >
> > On Thu, Mar 31, 2016 at 9:15 PM, Grant Henke 
> wrote:
> >
> > > Looking for some resolution on the "No Topics" change.
> > >
> > > I am thinking that using null in the protocol isn't that complex, and
> it
> > > avoids various edge cases with having multiple fields. That leaves us
> > with
> > > 2 options:
> > >
> > >- A: null = no topics, empty = all topics
> > >- B: null = all topics, empty = no topics
> > >
> > > A is nice because it just adds new functionality, existing logic
> doesn't
> > > change
> > > B is nice because its more "intuitive", but has the drawback of
> changing
> > > what empty means from request v0
> > >
> > > I do not have a strong opinion on the approach taken, which makes me
> lean
> > > towards option A. Keep in mind at the user level, the apis in the
> various
> > > clients can map this however they like.
> > >
> > > Does anyone feel strongly about the choice?
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Thu, Mar 31, 2016 at 9:21 AM, Grant Henke 
> > wrote:
> > >
> > > > I had a second look at the proposed changes to Metadata Request and
> > > >> Response and it seems to me that having a `controller_id` field
> would
> > be
> > > >> more efficient for non-trivial cases than having a `is_controller`
> > field
> > > >>  for each broker (which would be false for all but 1 case).
> > > >
> > > >
> > > > I agree this is better. I will update it.
> > > >
> > > > Similar, but less clear is the best way to encode
> `marked_for_deletion`
> > > and
> > > >> `is_internal`. These will also be false for most topics (there is
> only
> > > one
> > > >> internal topic at the moment, for example), so it may make sense to
> > > have a
> > > >> topics_marked_for_deletion and internal_topics in the response.
> > Because
> > > >> topics are identified by strings, it is not as clear-cut as the
> > > >> controller_id case, but it still seems like it would be a win for
> when
> > > it
> > > >> matters most (when the number of topics is large).
> > > >>
> > > >
> > > > Thats an interesting idea. I can try making this change to see what
> it
> > > > would look like.
> > > >
> > > > Thanks,
> > > > Grant
> > > >
> > > > On Thu, Mar 31, 2016 at 8:59 AM, Ismael Juma 
> > wrote:
> > > >
> > > >> Hi Grant,
> > > >>
> > > >> I had a second look at the proposed changes to Metadata Request and
> > > >> Response and it seems to me that having a `controller_id` field
> would
> > be
> > > >> more efficient for non-trivial cases than having a `is_controller`
> > field
> > > >>  for each broker (which would be false for all but 1 case).
> > > >>
> > > >> Similar, but less clear is the best way to encode
> > `marked_for_deletion`
> > > >> and
> > > >> `is_internal`. These will also be false for most topics (there is
> only
> > > one
> > > >> internal topic at the moment, for example), so it may make sense to
> > > have a
> > > >> topics_marked_for_deletion and internal_topics in the response.
> > Because
> > > >> topics are identified by strings, it is not as clear-cut as the
> > > >> controller_id case, but it still seems like it would be a win for
> when
> > > it
> > > >> matters most (when the number of topics is large).
> > > >
> > > >
> > > >> Ismael
> > > >>
> > > >> On Mon, Mar 14, 2016 at 10:07 PM, Grant Henke 
> > > >> wrote:
> > > >>
> > > >> > I have been updating the KIP-4 wiki page based on the last KIP
> call
> > > and
> > > >> > wanted to get some review and discussion around the server side
> > > >> > implementation for admin requests. Both the "ideal" functionality
> > and
> > > >> the
> > > >> > "intermediated" functionality. The updates are still in progress,
> > but
> > > >> this
> > > >> > section is the most critical and will likely have the most
> > discussion.
> > > >> This
> > > >> > topic has had a few shifts in perspective and various discussions
> on
> > > >> > synchronous vs asynchronous

[jira] [Commented] (KAFKA-3338) Add print and writeAsText functions to the Streams DSL

2016-03-31 Thread Bill Bejeck (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15220957#comment-15220957
 ] 

Bill Bejeck commented on KAFKA-3338:


I'm wondering if we need to provide a Serde for the print/writeAsText 
functionality as Serdes are only required when reading from/writing
to a topic or when using a state store.

To give you an idea of where I'm coming from, here's the integration test for 
the implementation of print()
{code}
KStream transactionKStream =  
kStreamBuilder.stream(stringSerde,transactionSerde,"stocks");

transactionKStream.through(stringSerde, transactionSerde,"stocks-out").print()
  .map((k,v)-> new KeyValue<>(v.getSymbol(),v)).print()
  .aggregateByKey(StockTransactionCollector::new,
  (k, v, stockTransactionCollector) -> 
stockTransactionCollector.add(v),
   
TumblingWindows.of("stock-summaries").with(1),
   stringSerde,collectorSerde).print();
   .to(windowedSerde,collectorSerde,"transaction-summary");
{code}
This allows me to see all elements from each stream (in order) on stdout.

Thoughts?

> Add print and writeAsText functions to the Streams DSL
> --
>
> Key: KAFKA-3338
> URL: https://issues.apache.org/jira/browse/KAFKA-3338
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Guozhang Wang
>Assignee: Bill Bejeck
>  Labels: newbie++
> Fix For: 0.10.1.0
>
>
> We want to provide some REPL-like pattern for users for better debuggability. 
> More concretely, we want to allow users to easily inspect their intermediate 
> data streams in the topology while running the application. Theoretically 
> this can be done by using a break point, or by calling System.out.println() 
> inside the operator, or through a finer grained trace-level logging. But more 
> user-friendly API would be to add a print() function to the KStream / KTable 
> object like:
> {code}
> // Prints the elements in this stream to the stdout, i.e. "System.out" of the 
> JVM
> KStream#print(/* optional serde */);  
> KTable#print(/* optional serde */);  
> // Writes the stream as text file(s) to the specified location.
> KStream#writeAsText(String filePath, /* optional serde */);
> KTable#writeAsText(String filePath, /* optional serde */);
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: KIP-4 Wiki Update

2016-03-31 Thread Jason Gustafson

I also prefer B. In addition to being intuitive, it seems less error-prone
in the long term, though it might be a little annoying for clients
maintaining backwards compatibility.

-Jason

On Thu, Mar 31, 2016 at 1:24 PM, Ismael Juma  wrote:

> I prefer B, the fact that we version the protocol means that we can fix
> mistakes instead of living with them forever. We should take advantage of
> that.
>
> Ismael
>
> On Thu, Mar 31, 2016 at 9:15 PM, Grant Henke  wrote:
>
> > Looking for some resolution on the "No Topics" change.
> >
> > I am thinking that using null in the protocol isn't that complex, and it
> > avoids various edge cases with having multiple fields. That leaves us
> with
> > 2 options:
> >
> >- A: null = no topics, empty = all topics
> >- B: null = all topics, empty = no topics
> >
> > A is nice because it just adds new functionality, existing logic doesn't
> > change
> > B is nice because its more "intuitive", but has the drawback of changing
> > what empty means from request v0
> >
> > I do not have a strong opinion on the approach taken, which makes me lean
> > towards option A. Keep in mind at the user level, the apis in the various
> > clients can map this however they like.
> >
> > Does anyone feel strongly about the choice?
> >
> >
> >
> >
> >
> >
> >
> > On Thu, Mar 31, 2016 at 9:21 AM, Grant Henke 
> wrote:
> >
> > > I had a second look at the proposed changes to Metadata Request and
> > >> Response and it seems to me that having a `controller_id` field would
> be
> > >> more efficient for non-trivial cases than having a `is_controller`
> field
> > >>  for each broker (which would be false for all but 1 case).
> > >
> > >
> > > I agree this is better. I will update it.
> > >
> > > Similar, but less clear is the best way to encode `marked_for_deletion`
> > and
> > >> `is_internal`. These will also be false for most topics (there is only
> > one
> > >> internal topic at the moment, for example), so it may make sense to
> > have a
> > >> topics_marked_for_deletion and internal_topics in the response.
> Because
> > >> topics are identified by strings, it is not as clear-cut as the
> > >> controller_id case, but it still seems like it would be a win for when
> > it
> > >> matters most (when the number of topics is large).
> > >>
> > >
> > > Thats an interesting idea. I can try making this change to see what it
> > > would look like.
> > >
> > > Thanks,
> > > Grant
> > >
> > > On Thu, Mar 31, 2016 at 8:59 AM, Ismael Juma 
> wrote:
> > >
> > >> Hi Grant,
> > >>
> > >> I had a second look at the proposed changes to Metadata Request and
> > >> Response and it seems to me that having a `controller_id` field would
> be
> > >> more efficient for non-trivial cases than having a `is_controller`
> field
> > >>  for each broker (which would be false for all but 1 case).
> > >>
> > >> Similar, but less clear is the best way to encode
> `marked_for_deletion`
> > >> and
> > >> `is_internal`. These will also be false for most topics (there is only
> > one
> > >> internal topic at the moment, for example), so it may make sense to
> > have a
> > >> topics_marked_for_deletion and internal_topics in the response.
> Because
> > >> topics are identified by strings, it is not as clear-cut as the
> > >> controller_id case, but it still seems like it would be a win for when
> > it
> > >> matters most (when the number of topics is large).
> > >
> > >
> > >> Ismael
> > >>
> > >> On Mon, Mar 14, 2016 at 10:07 PM, Grant Henke 
> > >> wrote:
> > >>
> > >> > I have been updating the KIP-4 wiki page based on the last KIP call
> > and
> > >> > wanted to get some review and discussion around the server side
> > >> > implementation for admin requests. Both the "ideal" functionality
> and
> > >> the
> > >> > "intermediated" functionality. The updates are still in progress,
> but
> > >> this
> > >> > section is the most critical and will likely have the most
> discussion.
> > >> This
> > >> > topic has had a few shifts in perspective and various discussions on
> > >> > synchronous vs asynchronous server support. The wiki contains my
> > current
> > >> > perspective on the challenges and approach.
> > >> >
> > >> > If you have any thoughts or feedback on the "Server-side Admin
> Request
> > >> > handlers" section here
> > >> > <
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-2.Server-sideAdminRequesthandlers
> > >> > >.
> > >> > Lets discuss them in this thread.
> > >> >
> > >> > For reference the last KIP discussion can be viewed here:
> > >> > https://youtu.be/rFW0-zJqg5I?t=12m30s
> > >> >
> > >> > Thank you,
> > >> > Grant
> > >> > --
> > >> > Grant Henke
> > >> > Software Engineer | Cloudera
> > >> > gr...@cloudera.com | twitter.com/gchenke |
> linkedin.com/in/granthenke
> >

[GitHub] kafka pull request: [Minor]: Clean up of SourceTaskOffsetCommiter

2016-03-31 Thread Ishiihara

GitHub user Ishiihara opened a pull request:

https://github.com/apache/kafka/pull/1170

[Minor]: Clean up of SourceTaskOffsetCommiter



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Ishiihara/kafka minor-cleanup

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1170.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1170


commit 90d868bb383c62f732e3a68d7ed2eb98dcf39dbd
Author: Liquan Pei 
Date:   2016-03-31T23:21:29Z

Minor clean up of SourceTaskOffsetCommiter




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [VOTE] KIP-50 - Enhance Authorizer interface to be aware of supported Principal Types

2016-03-31 Thread parth brahmbhatt

+1

Thanks
Parth

On Wed, Mar 30, 2016 at 8:22 PM, Ewen Cheslack-Postava 
wrote:

> +1
>
> -Ewen
>
> On Wed, Mar 30, 2016 at 7:20 PM, Gwen Shapira  wrote:
>
> > So my +1 is back :)
> >
> > On Wed, Mar 30, 2016 at 4:51 PM, Ashish Singh 
> wrote:
> >
> > > My bad, I moved the config option to rejected alternatives. The new
> > method
> > > getSupportedPrincipalTypes is what the KIP is proposing.
> > > 
> > >
> > > On Wed, Mar 30, 2016 at 4:43 PM, Gwen Shapira 
> wrote:
> > >
> > > > My appologies, but I need to retract my vote.
> > > >
> > > > The KIP currently specifies two alternatives (none rejected). So I'm
> > not
> > > > sure what I'm voting on...
> > > >
> > > > For reference, my original +1 was for the new method:
> > > > getSupportedPrincipalTypes()
> > > > I'm not even sure how would a config work in that case.
> > > >
> > > > Anyway, we need to resolve this before voting.
> > > >
> > > > Gwen
> > > >
> > > > On Wed, Mar 30, 2016 at 4:34 PM, Gwen Shapira 
> > wrote:
> > > >
> > > > > +1.
> > > > >
> > > > > It's a small change and I can see how this will help improve
> > usability
> > > > for
> > > > > some of the authorizers.
> > > > >
> > > > > Gwen
> > > > >
> > > > > On Wed, Mar 30, 2016 at 11:02 AM, Ashish Singh <
> asi...@cloudera.com>
> > > > > wrote:
> > > > >
> > > > >> Hi Guys,
> > > > >>
> > > > >> I would like to open the vote for KIP-50.
> > > > >>
> > > > >> KIP:
> > > > >>
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-50+-+Enhance+Authorizer+interface+to+be+aware+of+supported+Principal+Types
> > > > >>
> > > > >> Discuss thread: here
> > > > >> <
> > > > >>
> > > >
> > >
> >
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201603.mbox/%3CCAGQG9cUCLDO0owdziDcL9iStXNF1wURyVNcEZedQJg%3DUuC7j%3DQ%40mail.gmail.com%3E
> > > > >> >
> > > > >>
> > > > >> JIRA: https://issues.apache.org/jira/browse/KAFKA-3186
> > > > >>
> > > > >> PR: https://github.com/apache/kafka/pull/861
> > > > >> 
> > > > >> --
> > > > >>
> > > > >> Regards,
> > > > >> Ashish
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Regards,
> > > Ashish
> > >
> >
>
>
>
> --
> Thanks,
> Ewen
>

Jira

2016-03-31 Thread Matthias J. Sax

Hi,

I was browsing Kafka JIRAs and realized, that I cannot assign JIRAs to
myself. Can anyone give me credentials for this, please?

username: mjsax

Thanks. :)


-Matthias



signature.asc
Description: OpenPGP digital signature

Re: [DISCUSS] KIP-35 - Retrieve protocol version

2016-03-31 Thread Ashish Singh

I agree with Jason. The blocking factor has been how to use the proposed
changes in java client without making it super complicated. With Jason's
suggestion we can get past this blocker, while keeping the core of the
proposal intact.


On Thu, Mar 31, 2016 at 2:51 PM, Jason Gustafson  wrote:

> Bumping this thread. I talked with Ashish and Magnus about this KIP offline
> and I'm gradually coming over. The new API actually stands by itself
> outside of the discussion about whether the client should support backwards
> compatibility or not. For the Java client, we could continue to support the
> current compatibility approach in which the client supports only brokers
> with the same version or greater. In that case, we would use this API only
> to assert that the current API versions are all supported, and raise an
> exception if they are not. This gives us the capability going forward to
> detect when the client is talking to an older broker, which we don't have
> right now. This check should be straightforward, so we could do it now,
> which would resolve some of the uneasiness about having an unused feature
> which we depended on other clients to test for us. Does that make sense or
> not?
>
> -Jason
>
> On Thu, Mar 17, 2016 at 4:06 PM, Ashish Singh  wrote:
>
> > We have proposed and discussed majorly three approaches so far, there
> were
> > many minor versions with small variations. Comparing them really
> requires a
> > side by side proposal and their pros/cons, and I agree with others that
> > this has been lacking in the KIP. We just updated the KIP with following
> > details.
> >
> > 1. Provide proposed changes in all the three proposals we have discussed
> so
> > far. Except the current proposal, these proposals are in rejected
> > alternatives.
> > 2. Provide reasoning on why the rejected proposals were rejected.
> > 3. Add scenarios for all of these proposals from a client developer and
> > core Kafka developer point of view.
> >
> > As we are really close to 0.10 deadline, a quick round of voting will
> > really help. If you really do not like the idea, please feel free to say
> > so. If the vote fails for the current proposal, it can at lease provide
> > recommendations that we should consider for next version of proposal and
> > put it up for vote again for next release. However, as stated earlier by
> > multiple people having this ASAP will be awesome.
> >
> > On Thu, Mar 17, 2016 at 3:29 PM, Dana Powers 
> > wrote:
> >
> > > On Thu, Mar 17, 2016 at 1:42 PM, Gwen Shapira 
> wrote:
> > >
> > > > "I think I would make this approach work by looking at the released
> > > server
> > > > version documentation for each version that I am trying to support
> and
> > > test
> > > > against*, manually identify the expected "protocol vectors" each
> > > supports,
> > > > store that as a map of vectors to "broker versions", check each
> vector
> > at
> > > > runtime until I find a match, and write code compatibility checks
> from
> > > > there."
> > > >
> > > > How is this better than a global version ID?
> > >
> > >
> > > As a client developer, it seems roughly the same. I think it probably
> > > avoids the server development workflow issues, and possibly the need to
> > > agree on semantics of the global version ID? But others surely are more
> > > qualified than me to comment on that part.
> > >
> > > -Dana
> > >
> >
> >
> >
> > --
> >
> > Regards,
> > Ashish
> >
>



-- 

Regards,
Ashish

Re: [DISCUSS] KIP-35 - Retrieve protocol version

2016-03-31 Thread Jason Gustafson

Bumping this thread. I talked with Ashish and Magnus about this KIP offline
and I'm gradually coming over. The new API actually stands by itself
outside of the discussion about whether the client should support backwards
compatibility or not. For the Java client, we could continue to support the
current compatibility approach in which the client supports only brokers
with the same version or greater. In that case, we would use this API only
to assert that the current API versions are all supported, and raise an
exception if they are not. This gives us the capability going forward to
detect when the client is talking to an older broker, which we don't have
right now. This check should be straightforward, so we could do it now,
which would resolve some of the uneasiness about having an unused feature
which we depended on other clients to test for us. Does that make sense or
not?

-Jason

On Thu, Mar 17, 2016 at 4:06 PM, Ashish Singh  wrote:

> We have proposed and discussed majorly three approaches so far, there were
> many minor versions with small variations. Comparing them really requires a
> side by side proposal and their pros/cons, and I agree with others that
> this has been lacking in the KIP. We just updated the KIP with following
> details.
>
> 1. Provide proposed changes in all the three proposals we have discussed so
> far. Except the current proposal, these proposals are in rejected
> alternatives.
> 2. Provide reasoning on why the rejected proposals were rejected.
> 3. Add scenarios for all of these proposals from a client developer and
> core Kafka developer point of view.
>
> As we are really close to 0.10 deadline, a quick round of voting will
> really help. If you really do not like the idea, please feel free to say
> so. If the vote fails for the current proposal, it can at lease provide
> recommendations that we should consider for next version of proposal and
> put it up for vote again for next release. However, as stated earlier by
> multiple people having this ASAP will be awesome.
>
> On Thu, Mar 17, 2016 at 3:29 PM, Dana Powers 
> wrote:
>
> > On Thu, Mar 17, 2016 at 1:42 PM, Gwen Shapira  wrote:
> >
> > > "I think I would make this approach work by looking at the released
> > server
> > > version documentation for each version that I am trying to support and
> > test
> > > against*, manually identify the expected "protocol vectors" each
> > supports,
> > > store that as a map of vectors to "broker versions", check each vector
> at
> > > runtime until I find a match, and write code compatibility checks from
> > > there."
> > >
> > > How is this better than a global version ID?
> >
> >
> > As a client developer, it seems roughly the same. I think it probably
> > avoids the server development workflow issues, and possibly the need to
> > agree on semantics of the global version ID? But others surely are more
> > qualified than me to comment on that part.
> >
> > -Dana
> >
>
>
>
> --
>
> Regards,
> Ashish
>

[jira] [Commented] (KAFKA-1981) Make log compaction point configurable

2016-03-31 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15220691#comment-15220691
 ] 

ASF GitHub Bot commented on KAFKA-1981:
---

GitHub user ewasserman reopened a pull request:

https://github.com/apache/kafka/pull/1168

KAFKA-1981 Make log compaction point configurable

@jkreps 

Implements control over the portion of the head of the log that will not be 
compacted (i.e. preserved in detail). 

The log cleaner can be configured retain a minimum amount of the 
uncompacted "head" of the log. 
This is enabled by setting one or more of the compaction lags:

log.cleaner.min.compaction.lag.ms
log.cleaner.min.compaction.lag.bytes
log.cleaner.min.compaction.lag.messages

with similar per topic configurations:

min.compaction.lag.ms
min.compaction.lag.bytes
min.compaction.lag.messages

These can be used to set constraints on the minimum message age, aggregate 
size, and/or count respectively that may be compacted. If none are set, all log 
segments are eligible for compaction except for the last segment, i.e. the one 
currently being written to. The active segment will not be compacted even if 
all of the compaction lag constraints are satisfied.

In particular this allows for the example use case described in the JIRA: 
"any consumer that is no more than 1 hour behind will get every message."

This contribution is my (Eric Wasserman's) original work and I license the 
work to the Kafka project under its open source license.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ewasserman/kafka feat-compaction-lag

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1168.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1168


commit d9fa5ca1c6b2f9e08689697b85b9b54a90afa67c
Author: Eric Wasserman 
Date:   2016-03-15T00:57:07Z

log

commit b9d2752b6db36a7d9464e8b4c12ec0d9cd8dfcf7
Author: Eric Wasserman 
Date:   2016-03-16T03:46:07Z

add compaction lag

commit dbb57bcf4c1d0fdf5c15c7f09838a6f47f64bd29
Author: Eric Wasserman 
Date:   2016-03-17T00:56:57Z

tests

commit ffa37a2d296bceb612152316ac163d67e1d71fe0
Author: Eric Wasserman 
Date:   2016-03-17T20:04:25Z

clean up test

commit f0536ae2612338460f07377afb388b64a84c7972
Author: Eric Wasserman 
Date:   2016-03-18T00:54:13Z

integration test with lag

commit 513a43296b99df2d0426abe9d09f942a95e02274
Author: Eric Wasserman 
Date:   2016-03-22T00:56:02Z

lag integration test

commit 7746924e1fb5d964fe609e941fec5611828e8720
Author: Eric Wasserman 
Date:   2016-03-23T17:22:08Z

add size and message count properties; update property names

commit 8ae2a1daf0cd970bc38d8654f73f9175ee93433e
Author: Eric Wasserman 
Date:   2016-03-23T23:36:39Z

final tests

commit 7b41064eca6a45ad1f4a74c2ad0e3d6cec8f2c25
Author: Eric Wasserman 
Date:   2016-03-23T23:37:33Z

generalized compaction lag

commit b5c247be267918f3c4f1163d81e95fb93cd6dc01
Author: Eric Wasserman 
Date:   2016-03-24T00:45:32Z

update documentation

commit 118fa196f41a8fa0198c672288433cfce30a322c
Author: Eric Wasserman 
Date:   2016-03-24T00:48:35Z

add missing license comment

commit fa0b96e984a327968824894859ae9c97e4d0ed0d
Author: Eric Wasserman 
Date:   2016-03-24T16:44:39Z

reverse log4j

commit c4c0d97cb02104e8aea73df6bc3305914768192a
Author: Eric Wasserman 
Date:   2016-03-16T03:46:07Z

add compaction lag

commit 45a02f90aeb378ccd51e25dac4b19a1bfabfd79b
Author: Eric Wasserman 
Date:   2016-03-24T23:09:46Z

Merge branch 'feat-compaction-lag' of github.com:ewasserman/kafka into 
feat-compaction-lag

commit 49051027872443d2a38141893dab2d6d68b7c310
Author: Eric Wasserman 
Date:   2016-03-30T23:10:33Z

Merge branch 'trunk' into feat-compaction-lag

commit c56ccd517ad3d414c4aee891975693cffc794359
Author: Eric Wasserman 
Date:   2016-03-30T23:37:00Z

adapt to changes in test utils.

commit 63601ea403bb7c91623127a683f409c19e0259a5
Author: Eric Wasserman 
Date:   2016-03-31T17:46:28Z

Merge branch 'trunk' into feat-compaction-lag




> Make log compaction point configurable
> --
>
> Key: KAFKA-1981
>

[GitHub] kafka pull request: KAFKA-1981 Make log compaction point configura...

2016-03-31 Thread ewasserman

GitHub user ewasserman reopened a pull request:

https://github.com/apache/kafka/pull/1168

KAFKA-1981 Make log compaction point configurable

@jkreps 

Implements control over the portion of the head of the log that will not be 
compacted (i.e. preserved in detail). 

The log cleaner can be configured retain a minimum amount of the 
uncompacted "head" of the log. 
This is enabled by setting one or more of the compaction lags:

log.cleaner.min.compaction.lag.ms
log.cleaner.min.compaction.lag.bytes
log.cleaner.min.compaction.lag.messages

with similar per topic configurations:

min.compaction.lag.ms
min.compaction.lag.bytes
min.compaction.lag.messages

These can be used to set constraints on the minimum message age, aggregate 
size, and/or count respectively that may be compacted. If none are set, all log 
segments are eligible for compaction except for the last segment, i.e. the one 
currently being written to. The active segment will not be compacted even if 
all of the compaction lag constraints are satisfied.

In particular this allows for the example use case described in the JIRA: 
"any consumer that is no more than 1 hour behind will get every message."

This contribution is my (Eric Wasserman's) original work and I license the 
work to the Kafka project under its open source license.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ewasserman/kafka feat-compaction-lag

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1168.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1168


commit d9fa5ca1c6b2f9e08689697b85b9b54a90afa67c
Author: Eric Wasserman 
Date:   2016-03-15T00:57:07Z

log

commit b9d2752b6db36a7d9464e8b4c12ec0d9cd8dfcf7
Author: Eric Wasserman 
Date:   2016-03-16T03:46:07Z

add compaction lag

commit dbb57bcf4c1d0fdf5c15c7f09838a6f47f64bd29
Author: Eric Wasserman 
Date:   2016-03-17T00:56:57Z

tests

commit ffa37a2d296bceb612152316ac163d67e1d71fe0
Author: Eric Wasserman 
Date:   2016-03-17T20:04:25Z

clean up test

commit f0536ae2612338460f07377afb388b64a84c7972
Author: Eric Wasserman 
Date:   2016-03-18T00:54:13Z

integration test with lag

commit 513a43296b99df2d0426abe9d09f942a95e02274
Author: Eric Wasserman 
Date:   2016-03-22T00:56:02Z

lag integration test

commit 7746924e1fb5d964fe609e941fec5611828e8720
Author: Eric Wasserman 
Date:   2016-03-23T17:22:08Z

add size and message count properties; update property names

commit 8ae2a1daf0cd970bc38d8654f73f9175ee93433e
Author: Eric Wasserman 
Date:   2016-03-23T23:36:39Z

final tests

commit 7b41064eca6a45ad1f4a74c2ad0e3d6cec8f2c25
Author: Eric Wasserman 
Date:   2016-03-23T23:37:33Z

generalized compaction lag

commit b5c247be267918f3c4f1163d81e95fb93cd6dc01
Author: Eric Wasserman 
Date:   2016-03-24T00:45:32Z

update documentation

commit 118fa196f41a8fa0198c672288433cfce30a322c
Author: Eric Wasserman 
Date:   2016-03-24T00:48:35Z

add missing license comment

commit fa0b96e984a327968824894859ae9c97e4d0ed0d
Author: Eric Wasserman 
Date:   2016-03-24T16:44:39Z

reverse log4j

commit c4c0d97cb02104e8aea73df6bc3305914768192a
Author: Eric Wasserman 
Date:   2016-03-16T03:46:07Z

add compaction lag

commit 45a02f90aeb378ccd51e25dac4b19a1bfabfd79b
Author: Eric Wasserman 
Date:   2016-03-24T23:09:46Z

Merge branch 'feat-compaction-lag' of github.com:ewasserman/kafka into 
feat-compaction-lag

commit 49051027872443d2a38141893dab2d6d68b7c310
Author: Eric Wasserman 
Date:   2016-03-30T23:10:33Z

Merge branch 'trunk' into feat-compaction-lag

commit c56ccd517ad3d414c4aee891975693cffc794359
Author: Eric Wasserman 
Date:   2016-03-30T23:37:00Z

adapt to changes in test utils.

commit 63601ea403bb7c91623127a683f409c19e0259a5
Author: Eric Wasserman 
Date:   2016-03-31T17:46:28Z

Merge branch 'trunk' into feat-compaction-lag




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-1981) Make log compaction point configurable

2016-03-31 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15220689#comment-15220689
 ] 

ASF GitHub Bot commented on KAFKA-1981:
---

Github user ewasserman closed the pull request at:

https://github.com/apache/kafka/pull/1168


> Make log compaction point configurable
> --
>
> Key: KAFKA-1981
> URL: https://issues.apache.org/jira/browse/KAFKA-1981
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.2.0
>Reporter: Jay Kreps
>  Labels: newbie++
>
> Currently if you enable log compaction the compactor will kick in whenever 
> you hit a certain "dirty ratio", i.e. when 50% of your data is uncompacted. 
> Other than this we don't give you fine-grained control over when compaction 
> occurs. In addition we never compact the active segment (since it is still 
> being written to).
> Other than this we don't really give you much control over when compaction 
> will happen. The result is that you can't really guarantee that a consumer 
> will get every update to a compacted topic--if the consumer falls behind a 
> bit it might just get the compacted version.
> This is usually fine, but it would be nice to make this more configurable so 
> you could set either a # messages, size, or time bound for compaction.
> This would let you say, for example, "any consumer that is no more than 1 
> hour behind will get every message."
> This should be relatively easy to implement since it just impacts the 
> end-point the compactor considers available for compaction. I think we 
> already have that concept, so this would just be some other overrides to add 
> in when calculating that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-1981 Make log compaction point configura...

2016-03-31 Thread ewasserman

Github user ewasserman closed the pull request at:

https://github.com/apache/kafka/pull/1168


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [VOTE] 0.10.0.0 RC1

2016-03-31 Thread Becket Qin

I am not sure about the other KIPs, For KIP-33 and KIP-47 I think it is
enough if we have reasonable review iterations.

On Thu, Mar 31, 2016 at 10:54 AM, Gwen Shapira  wrote:

> Good question :)
>
> Lets say we want:
> KIP-43, KIP-35, KIP-4-metadata and KIP-33 to get in. How much time do we
> need?
>
> I think 2 or 3 weeks is reasonable? Am I being too optimistic again?
>
> Gwen
>
>
> On Thu, Mar 31, 2016 at 10:42 AM, Becket Qin  wrote:
>
> > Sure, do we have a new intended release close date?
> >
> > On Thu, Mar 31, 2016 at 10:23 AM, Gwen Shapira 
> wrote:
> >
> > > I'd LOVE to have KIP-33 get it.
> > >
> > > Can you work with Jun to make sure the timing will work?
> > >
> > > On Thu, Mar 31, 2016 at 9:20 AM, Becket Qin 
> > wrote:
> > >
> > > > Hi Gwen,
> > > >
> > > > KIP-47 is voted and passed, but not merged yet. It is actually
> > depending
> > > on
> > > > KIP-33 which is in the voting process.
> > > >
> > > > I know we discussed on the KIP hangout that we will do KIP-33 post
> > > 0.10.0.
> > > > But since 0.10.0 is delayed, maybe we can include KIP-33 given it is
> a
> > > > long-wanted fix by many users.
> > > > And I agree with Grant that it would be nice to have KIP-35 in this
> > > > release.
> > > >
> > > > Thanks,
> > > >
> > > > Jiangjie (Becket) Qin
> > > >
> > > > On Wed, Mar 30, 2016 at 2:35 PM, Gwen Shapira 
> > wrote:
> > > >
> > > > > I think we are in agreement here.
> > > > > Note that KIP-47 is already voted and merged AFAIK, so it will be
> in
> > > > 0.10.0
> > > > > either way (actually, it will be nice if someone tries using this
> > > feature
> > > > > on a the RC...)
> > > > >
> > > > > Gwen
> > > > >
> > > > > On Wed, Mar 30, 2016 at 11:53 AM, Grant Henke  >
> > > > wrote:
> > > > >
> > > > > > Below are my thoughts on the release features. I don't feel too
> > > > strongly
> > > > > > about it, but I figured I would lay out my current perspective on
> > the
> > > > > 0.10
> > > > > > release.
> > > > > >
> > > > > > When considering the optimal time to do this major release the
> > > > > > considerations I am weighing are:
> > > > > >
> > > > > >- Compatibility
> > > > > >   - Are more breaking changes almost ready? We want major
> > > releases
> > > > to
> > > > > >   be infrequent. Having the breaking changes/pieces in will
> > help
> > > > 0.10
> > > > > > last
> > > > > >   longer.
> > > > > >   - Breaking again in 3 months is not favorable, especially
> for
> > > > > >   clients.
> > > > > >- Features vs Risk
> > > > > >   - Is it worth it for a user to upgrade? Given there is risk
> > in
> > > a
> > > > > >   major upgrade, are there enough features to take that risk
> > on?
> > > > > >   - The breaking change we have introduced is the timestamp
> > > changes
> > > > > >   (KIP-31/KIP-32), but users can't use it yet (KIP-33)
> beyond a
> > > > purge
> > > > > >   improvement.
> > > > > >   - We did add Streams but that could technically be added
> in a
> > > > 0.9.1
> > > > > >   release. (I think)
> > > > > >
> > > > > > For those reasons I would prefer to block on these changes for
> 0.10
> > > > > >
> > > > > >- KIP-4's Metadata changes
> > > > > >- Breaking wire protocol change
> > > > > >   - Also, fixes critical issues and adds rack fields from
> > KIP-36
> > > > > >   - Patch available, need consensus/vote on metadata protocol
> > > > change
> > > > > >- KIP-35: Retrieving protocol version
> > > > > >- Though Kafka's internal usage has not been fully vetted, we
> > all
> > > > > agree
> > > > > >   its useful to be able to ask for the protocol versions
> > > > > >   - The Kafka client community really wants this
> > > > > >   - It can reduce the impact/breakage of future releases
> > > > > >- KIP-33: Add a time based log index
> > > > > >   - This was a major motivation for KIP-31/32 and adding it
> > could
> > > > > help
> > > > > >   vet those changes to ensure we don't need a breaking change
> > > later
> > > > > >   - Adding a feature flag to disable it broker side has been
> > > > > discussed
> > > > > >   to mitigate risk
> > > > > >   - It doesn't need to block the release if we are confident
> it
> > > > won't
> > > > > >   need a breaking change
> > > > > >
> > > > > > It would also be nice to get these in if ready, but I don't think
> > the
> > > > > need
> > > > > > to block the release:
> > > > > >
> > > > > >- KIP-43: Kafka SASL enhancements
> > > > > >   - Scope has been reduced
> > > > > >   - Seams it could be added quickly
> > > > > >   - Not breaking addition, so doesn't need to block
> > > > > >- KIP-47: Add timestamp-based log deletion policy
> > > > > >- Also leverages KIP-32/33 further vetting its implementation
> > > > > >   - Not breaking addition, so doesn't need to

Re: KIP-4 Wiki Update

2016-03-31 Thread Ismael Juma

I prefer B, the fact that we version the protocol means that we can fix
mistakes instead of living with them forever. We should take advantage of
that.

Ismael

On Thu, Mar 31, 2016 at 9:15 PM, Grant Henke  wrote:

> Looking for some resolution on the "No Topics" change.
>
> I am thinking that using null in the protocol isn't that complex, and it
> avoids various edge cases with having multiple fields. That leaves us with
> 2 options:
>
>- A: null = no topics, empty = all topics
>- B: null = all topics, empty = no topics
>
> A is nice because it just adds new functionality, existing logic doesn't
> change
> B is nice because its more "intuitive", but has the drawback of changing
> what empty means from request v0
>
> I do not have a strong opinion on the approach taken, which makes me lean
> towards option A. Keep in mind at the user level, the apis in the various
> clients can map this however they like.
>
> Does anyone feel strongly about the choice?
>
>
>
>
>
>
>
> On Thu, Mar 31, 2016 at 9:21 AM, Grant Henke  wrote:
>
> > I had a second look at the proposed changes to Metadata Request and
> >> Response and it seems to me that having a `controller_id` field would be
> >> more efficient for non-trivial cases than having a `is_controller` field
> >>  for each broker (which would be false for all but 1 case).
> >
> >
> > I agree this is better. I will update it.
> >
> > Similar, but less clear is the best way to encode `marked_for_deletion`
> and
> >> `is_internal`. These will also be false for most topics (there is only
> one
> >> internal topic at the moment, for example), so it may make sense to
> have a
> >> topics_marked_for_deletion and internal_topics in the response. Because
> >> topics are identified by strings, it is not as clear-cut as the
> >> controller_id case, but it still seems like it would be a win for when
> it
> >> matters most (when the number of topics is large).
> >>
> >
> > Thats an interesting idea. I can try making this change to see what it
> > would look like.
> >
> > Thanks,
> > Grant
> >
> > On Thu, Mar 31, 2016 at 8:59 AM, Ismael Juma  wrote:
> >
> >> Hi Grant,
> >>
> >> I had a second look at the proposed changes to Metadata Request and
> >> Response and it seems to me that having a `controller_id` field would be
> >> more efficient for non-trivial cases than having a `is_controller` field
> >>  for each broker (which would be false for all but 1 case).
> >>
> >> Similar, but less clear is the best way to encode `marked_for_deletion`
> >> and
> >> `is_internal`. These will also be false for most topics (there is only
> one
> >> internal topic at the moment, for example), so it may make sense to
> have a
> >> topics_marked_for_deletion and internal_topics in the response. Because
> >> topics are identified by strings, it is not as clear-cut as the
> >> controller_id case, but it still seems like it would be a win for when
> it
> >> matters most (when the number of topics is large).
> >
> >
> >> Ismael
> >>
> >> On Mon, Mar 14, 2016 at 10:07 PM, Grant Henke 
> >> wrote:
> >>
> >> > I have been updating the KIP-4 wiki page based on the last KIP call
> and
> >> > wanted to get some review and discussion around the server side
> >> > implementation for admin requests. Both the "ideal" functionality and
> >> the
> >> > "intermediated" functionality. The updates are still in progress, but
> >> this
> >> > section is the most critical and will likely have the most discussion.
> >> This
> >> > topic has had a few shifts in perspective and various discussions on
> >> > synchronous vs asynchronous server support. The wiki contains my
> current
> >> > perspective on the challenges and approach.
> >> >
> >> > If you have any thoughts or feedback on the "Server-side Admin Request
> >> > handlers" section here
> >> > <
> >> >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-2.Server-sideAdminRequesthandlers
> >> > >.
> >> > Lets discuss them in this thread.
> >> >
> >> > For reference the last KIP discussion can be viewed here:
> >> > https://youtu.be/rFW0-zJqg5I?t=12m30s
> >> >
> >> > Thank you,
> >> > Grant
> >> > --
> >> > Grant Henke
> >> > Software Engineer | Cloudera
> >> > gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
> >> >
> >>
> >
> >
> >
> > --
> > Grant Henke
> > Software Engineer | Cloudera
> > gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
> >
>
>
>
> --
> Grant Henke
> Software Engineer | Cloudera
> gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
>

[jira] [Comment Edited] (KAFKA-3334) First message on new topic not actually being sent, no exception thrown

2016-03-31 Thread Jiangjie Qin (JIRA)

[
https://issues.apache.org/jira/browse/KAFKA-3334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15220604#comment-15220604
]

Jiangjie Qin edited comment on KAFKA-3334 at 3/31/16 8:21 PM:
--

[~singhashish] I think we are on the same page that we want to let user have a
clear idea about where to look at if something goes wrong. In terms of
documentation, it is probably extremely difficult to document all the possible
scenario user might see because we have so many different configuration
combinations and each combination might result in different behaviors.
Documentation based on scenario might never be enough:) I was thinking about
the following:

1. The default configurations should just work out of the box in general if
user does not change any configurations.
2. For each configuration, we need to document clearly what this configuration
is for and what are the possible impact.
3. For each exception thrown to user, a clear yet brief message about what was
wrong should be in the error message itself. In the documentation of the
exceptions, we can list the possible places this exception is thrown (this
should match the info in the error message) and what is the possible cause as
well as suggested solution.

Taking this particular case as an example, in the TimeoutException thrown from
producer.send() user will see
{noformat}
"The producer failed to fetch the metadata for the topic XXX after XXX ms.
Please see the exception documentation for possible cause."
{noformat}

And the documentation of TimeoutException should have something like
{noformat}
"This exception can be thrown in the following cases:
1. The producer cannot fetch the metadata of a topic. This only happens when
the producer is sending message to the topic for the first time. It is more
likely to happen if the topic did not exist on the brokers. The new topic
creation on the broker might take some time. User can retry send the message in
this case.
2. blah blah blah"
{noformat}

I feel this is more intuitive for the users to get an idea about what went
wrong because at the end of the day, the first thing user will see is the
exception. If the exception itself does not provide clear pointer, users do not
know where to start. For example, if user see TimeoutException, what are they
supposed to search or read?

So my point is that we should provide crystal clear message in the exception
itself, through both error message and documentation.

I agree that it might also be useful if we provide the detail on how
KafkaProducer sends the message. But it seems for users really care about the
internal details, reading the code is probably the best way.

was (Author: becket_qin):
[~singhashish] I think we are on the same page that we want to let user have a
clear idea where to look at if something goes wrong. In terms of documentation,
it is probably extremely difficult to document all the possible scenario user
might see because we have so many different configuration combinations and each
combination might result in different behaviors. Documentation based on
scenario might never be enough:) I was thinking about the following:

So my point is that we should provide crystal clear

[jira] [Commented] (KAFKA-3334) First message on new topic not actually being sent, no exception thrown

2016-03-31 Thread Jiangjie Qin (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15220604#comment-15220604
 ] 

Jiangjie Qin commented on KAFKA-3334:
-

[~singhashish] I think we are on the same page that we want to let user have a 
clear idea where to look at if something goes wrong. In terms of documentation, 
it is probably extremely difficult to document all the possible scenario user 
might see because we have so many different configuration combinations and each 
combination might result in different behaviors. Documentation based on 
scenario might never be enough:) I was thinking about the following:

1. The default configurations should just work out of the box in general if 
user does not change any configurations.
2. For each configuration, we need to document clearly what this configuration 
is for and what are the possible impact.
3. For each exception thrown to user, a clear yet brief message about what was 
wrong should be in the error message itself. In the documentation of the 
exceptions, we can list the possible places this exception is thrown (this 
should match the info in the error message) and what is the possible cause as 
well as suggested solution.

Taking this particular case as an example, in the TimeoutException thrown from 
producer.send() user will see 
{noformat}
"The producer failed to fetch the metadata for the topic XXX after XXX ms. 
Please see the exception documentation for possible cause."
{noformat}

And the documentation of TimeoutException should have something like 
{noformat}
"This exception can be thrown in the following cases:
1. The producer cannot fetch the metadata of a topic. This only happens when 
the producer is sending message to the topic for the first time. It is more 
likely to happen if the topic did not exist on the brokers. The new topic 
creation on the broker might take some time. User can retry send the message in 
this case.
2. blah blah blah"
{noformat}

I feel this is more intuitive for the users to get an idea about what went 
wrong because at the end of the day, the first thing user will see is the 
exception. If the exception itself does not provide clear pointer, users do not 
know where to start. For example, if user see TimeoutException, what are they 
supposed to search or read?

So my point is that we should provide crystal clear message in the exception 
itself, through both error message and documentation.

I agree that it might also be useful if we provide the detail on how 
KafkaProducer sends the message. But it seems for users really care about the 
internal details, reading the code is probably the best way.

> First message on new topic not actually being sent, no exception thrown
> ---
>
> Key: KAFKA-3334
> URL: https://issues.apache.org/jira/browse/KAFKA-3334
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.0
> Environment: Linux, Java
>Reporter: Aleksandar Stojadinovic
>Assignee: Ashish K Singh
> Fix For: 0.10.1.0
>
>
> Although I've seen this issue pop around the internet in a few forms, I'm not 
> sure it is yet properly fixed. 
> When publishing to a new topic, with auto create-enabled, the java client 
> (0.9.0) shows this WARN message in the log, and the message is not sent 
> obviously:
> org.apache.kafka.clients.NetworkClient - Error while fetching metadata with 
> correlation id 0 : {file.topic=LEADER_NOT_AVAILABLE}
> In the meantime I see in the console the message that a log for partition is 
> created. The next messages are patched through normally, but the first one is 
> never sent. No exception is ever thrown, either by calling get on the future, 
> or with the async usage, like everything is perfect.
> I notice when I leave my application blocked on the get call, in the 
> debugger, then the message may be processed, but with significant delay. This 
> is consistent with another issue I found for the python client. Also, if I 
> call partitionsFor previously, the topic is created and the message is sent. 
> But it seems silly to call it every time, just to mitigate this issue.
> {code}
> Future recordMetadataFuture = producer.send(new 
> ProducerRecord<>(topic, key, file));
> RecordMetadata recordMetadata = recordMetadataFuture.get(30, 
> TimeUnit.SECONDS);
> {code}
> I hope I'm clear enough.
> Related similar (but not same) issues:
> https://issues.apache.org/jira/browse/KAFKA-1124
> https://github.com/dpkp/kafka-python/issues/150
> http://stackoverflow.com/questions/35187933/how-to-resolve-leader-not-available-kafka-error-when-trying-to-consume



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: KIP-4 Wiki Update

2016-03-31 Thread Grant Henke

Looking for some resolution on the "No Topics" change.

I am thinking that using null in the protocol isn't that complex, and it
avoids various edge cases with having multiple fields. That leaves us with
2 options:

   - A: null = no topics, empty = all topics
   - B: null = all topics, empty = no topics

A is nice because it just adds new functionality, existing logic doesn't
change
B is nice because its more "intuitive", but has the drawback of changing
what empty means from request v0

I do not have a strong opinion on the approach taken, which makes me lean
towards option A. Keep in mind at the user level, the apis in the various
clients can map this however they like.

Does anyone feel strongly about the choice?







On Thu, Mar 31, 2016 at 9:21 AM, Grant Henke  wrote:

> I had a second look at the proposed changes to Metadata Request and
>> Response and it seems to me that having a `controller_id` field would be
>> more efficient for non-trivial cases than having a `is_controller` field
>>  for each broker (which would be false for all but 1 case).
>
>
> I agree this is better. I will update it.
>
> Similar, but less clear is the best way to encode `marked_for_deletion` and
>> `is_internal`. These will also be false for most topics (there is only one
>> internal topic at the moment, for example), so it may make sense to have a
>> topics_marked_for_deletion and internal_topics in the response. Because
>> topics are identified by strings, it is not as clear-cut as the
>> controller_id case, but it still seems like it would be a win for when it
>> matters most (when the number of topics is large).
>>
>
> Thats an interesting idea. I can try making this change to see what it
> would look like.
>
> Thanks,
> Grant
>
> On Thu, Mar 31, 2016 at 8:59 AM, Ismael Juma  wrote:
>
>> Hi Grant,
>>
>> I had a second look at the proposed changes to Metadata Request and
>> Response and it seems to me that having a `controller_id` field would be
>> more efficient for non-trivial cases than having a `is_controller` field
>>  for each broker (which would be false for all but 1 case).
>>
>> Similar, but less clear is the best way to encode `marked_for_deletion`
>> and
>> `is_internal`. These will also be false for most topics (there is only one
>> internal topic at the moment, for example), so it may make sense to have a
>> topics_marked_for_deletion and internal_topics in the response. Because
>> topics are identified by strings, it is not as clear-cut as the
>> controller_id case, but it still seems like it would be a win for when it
>> matters most (when the number of topics is large).
>
>
>> Ismael
>>
>> On Mon, Mar 14, 2016 at 10:07 PM, Grant Henke 
>> wrote:
>>
>> > I have been updating the KIP-4 wiki page based on the last KIP call and
>> > wanted to get some review and discussion around the server side
>> > implementation for admin requests. Both the "ideal" functionality and
>> the
>> > "intermediated" functionality. The updates are still in progress, but
>> this
>> > section is the most critical and will likely have the most discussion.
>> This
>> > topic has had a few shifts in perspective and various discussions on
>> > synchronous vs asynchronous server support. The wiki contains my current
>> > perspective on the challenges and approach.
>> >
>> > If you have any thoughts or feedback on the "Server-side Admin Request
>> > handlers" section here
>> > <
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-2.Server-sideAdminRequesthandlers
>> > >.
>> > Lets discuss them in this thread.
>> >
>> > For reference the last KIP discussion can be viewed here:
>> > https://youtu.be/rFW0-zJqg5I?t=12m30s
>> >
>> > Thank you,
>> > Grant
>> > --
>> > Grant Henke
>> > Software Engineer | Cloudera
>> > gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
>> >
>>
>
>
>
> --
> Grant Henke
> Software Engineer | Cloudera
> gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
>



-- 
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke

Build failed in Jenkins: kafka-trunk-jdk7 #1162

2016-03-31 Thread Apache Jenkins Server

See 

Changes:

[cshapi] KAFKA-3451: Add basic HTML coverage report generation to gradle

--
[...truncated 1600 lines...]

kafka.log.LogTest > testTruncateTo PASSED

kafka.log.LogTest > testCleanShutdownFile PASSED

kafka.log.OffsetIndexTest > lookupExtremeCases PASSED

kafka.log.OffsetIndexTest > appendTooMany PASSED

kafka.log.OffsetIndexTest > randomLookupTest PASSED

kafka.log.OffsetIndexTest > testReopen PASSED

kafka.log.OffsetIndexTest > appendOutOfOrder PASSED

kafka.log.OffsetIndexTest > truncate PASSED

kafka.log.LogSegmentTest > testRecoveryWithCorruptMessage PASSED

kafka.log.LogSegmentTest > testRecoveryFixesCorruptIndex PASSED

kafka.log.LogSegmentTest > testReadFromGap PASSED

kafka.log.LogSegmentTest > testTruncate PASSED

kafka.log.LogSegmentTest > testReadBeforeFirstOffset PASSED

kafka.log.LogSegmentTest > testCreateWithInitFileSizeAppendMessage PASSED

kafka.log.LogSegmentTest > testChangeFileSuffixes PASSED

kafka.log.LogSegmentTest > testMaxOffset PASSED

kafka.log.LogSegmentTest > testNextOffsetCalculation PASSED

kafka.log.LogSegmentTest > testReadOnEmptySegment PASSED

kafka.log.LogSegmentTest > testReadAfterLast PASSED

kafka.log.LogSegmentTest > testCreateWithInitFileSizeClearShutdown PASSED

kafka.log.LogSegmentTest > testTruncateFull PASSED

kafka.log.CleanerTest > testBuildOffsetMap PASSED

kafka.log.CleanerTest > testSegmentGrouping PASSED

kafka.log.CleanerTest > testCleanSegmentsWithAbort PASSED

kafka.log.CleanerTest > testSegmentGroupingWithSparseOffsets PASSED

kafka.log.CleanerTest > testRecoveryAfterCrash PASSED

kafka.log.CleanerTest > testLogToClean PASSED

kafka.log.CleanerTest > testCleaningWithDeletes PASSED

kafka.log.CleanerTest > testCleanSegments PASSED

kafka.log.CleanerTest > testCleaningWithUnkeyedMessages PASSED

kafka.log.OffsetMapTest > testClear PASSED

kafka.log.OffsetMapTest > testBasicValidation PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testHeartbeatWrongCoordinator 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testDescribeGroupStable PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testHeartbeatIllegalGeneration 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testDescribeGroupWrongCoordinator PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testDescribeGroupRebalancing 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testLeaderFailureInSyncGroup 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testGenerationIdIncrementsOnRebalance PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testSyncGroupFromIllegalGeneration PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testInvalidGroupId PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testHeartbeatUnknownGroup 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testListGroupsIncludesStableGroups PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testHeartbeatDuringRebalanceCausesRebalanceInProgress PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupInconsistentGroupProtocol PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupSessionTimeoutTooLarge PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupSessionTimeoutTooSmall PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testSyncGroupEmptyAssignment 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testCommitOffsetWithDefaultGeneration PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupFromUnchangedLeaderShouldRebalance PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testHeartbeatRebalanceInProgress PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testLeaveGroupUnknownGroup 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testListGroupsIncludesRebalancingGroups PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testSyncGroupFollowerAfterLeader PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testCommitOffsetInAwaitingSync 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testJoinGroupWrongCoordinator 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupUnknownConsumerExistingGroup PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testSyncGroupFromUnknownGroup 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupInconsistentProtocolType PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testCommitOffsetFromUnknownGroup PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testLeaveGroupWrongCoordinator 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testLeaveGroupUnknownConsumerExistingGroup PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupUnknownConsumerNewGroup PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupFromUnchangedFollowerDoesNotRebalance PASSED

[jira] [Commented] (KAFKA-3488) commitAsync() fails if metadata update creates new SASL/SSL connection

2016-03-31 Thread Rajini Sivaram (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15220416#comment-15220416
 ] 

Rajini Sivaram commented on KAFKA-3488:
---

[~hachikuji] Thank you for the feedback. I was thinking along the lines of 2) 
since it felt like the simplest change. I will take a look at the commits to 
see if I can reuse the code. I will look into 1) as well. Thanks.

> commitAsync() fails if metadata update creates new SASL/SSL connection
> --
>
> Key: KAFKA-3488
> URL: https://issues.apache.org/jira/browse/KAFKA-3488
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.9.0.1
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
> Fix For: 0.10.0.0
>
>
> Sasl/SslConsumerTest.testSimpleConsumption() fails intermittently with a 
> failure in {{commitAsync()}}. The exception stack trace shows:
> {quote}
> kafka.api.SaslPlaintextConsumerTest.testSimpleConsumption FAILED
> java.lang.AssertionError: expected:<1> but was:<0>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:834)
>   at org.junit.Assert.assertEquals(Assert.java:645)
>   at org.junit.Assert.assertEquals(Assert.java:631)
>   at 
> kafka.api.BaseConsumerTest.awaitCommitCallback(BaseConsumerTest.scala:340)
>   at 
> kafka.api.BaseConsumerTest.testSimpleConsumption(BaseConsumerTest.scala:85)
> {quote}
> I have recreated this with some additional trace. The tests run with a very 
> small metadata expiry interval, triggering metadata updates quite often. If a 
> metadata request immediately following a {{commitAsync()}} call creates a new 
> SSL/SASL connection, {{ConsumerNetworkClient.poll}} returns to process the 
> connection handshake packets. Since {{ConsumerNetworkClient.poll}} discards 
> all unsent packets before returning from poll, this can result in the failure 
> of the commit - the callback is invoked with {{SendFailedException}}.
> I understand that {{ConsumerNetworkClient.poll()}} discards unsent packets 
> rather than buffer them to keep the code simple. And perhaps it is ok to fail 
> {{commitAsync}} occasionally since the callback does indicate that the caller 
> should retry. But it feels like an unnecessary limitation that requires error 
> handling in client applications when there are no real failures and makes it 
> much harder to test reliably. As special handling to fix issues like 
> KAFKA-3412, KAFKA-2672 adds more complexity to the code anyway, and because 
> it is much harder to debug failures that affect only SSL/SASL, it may be 
> worth considering improving this behaviour.
> I will see if I can submit a PR for the specific issue I was seeing with the 
> impact of handshakes on {{commitAsync()}}, but I will be interested in views 
> on improving the logic in {{ConsumerNetworkClient}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3469) kafka-topics lock down znodes with user principal when zk security is enabled.

2016-03-31 Thread Ashish K Singh (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15220360#comment-15220360
 ] 

Ashish K Singh commented on KAFKA-3469:
---

[~fpj] pinging again for your feedback.

> kafka-topics lock down znodes with user principal when zk security is enabled.
> --
>
> Key: KAFKA-3469
> URL: https://issues.apache.org/jira/browse/KAFKA-3469
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ashish K Singh
>Assignee: Ashish K Singh
>
> In envs where ZK is kerberized, if a user, other than user running kafka 
> processes, creates a topic, ZkUtils will lock down corresponding znodes for 
> the user. Kafka will not be able to modify those znodes and that leaves the 
> topic unusable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-0.10.0-jdk7 #25

2016-03-31 Thread Apache Jenkins Server

See 

Changes:

[cshapi] KAFKA-3451: Add basic HTML coverage report generation to gradle

--
[...truncated 1565 lines...]

kafka.log.BrokerCompressionTest > testBrokerSideCompression[17] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[18] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[19] PASSED

kafka.log.LogSegmentTest > testRecoveryWithCorruptMessage PASSED

kafka.log.LogSegmentTest > testRecoveryFixesCorruptIndex PASSED

kafka.log.LogSegmentTest > testReadFromGap PASSED

kafka.log.LogSegmentTest > testTruncate PASSED

kafka.log.LogSegmentTest > testReadBeforeFirstOffset PASSED

kafka.log.LogSegmentTest > testCreateWithInitFileSizeAppendMessage PASSED

kafka.log.LogSegmentTest > testChangeFileSuffixes PASSED

kafka.log.LogSegmentTest > testMaxOffset PASSED

kafka.log.LogSegmentTest > testNextOffsetCalculation PASSED

kafka.log.LogSegmentTest > testReadOnEmptySegment PASSED

kafka.log.LogSegmentTest > testReadAfterLast PASSED

kafka.log.LogSegmentTest > testCreateWithInitFileSizeClearShutdown PASSED

kafka.log.LogSegmentTest > testTruncateFull PASSED

kafka.log.LogConfigTest > testFromPropsEmpty PASSED

kafka.log.LogConfigTest > testKafkaConfigToProps PASSED

kafka.log.LogConfigTest > testFromPropsInvalid PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[0] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[1] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[2] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[3] PASSED

kafka.log.LogManagerTest > testCleanupSegmentsToMaintainSize PASSED

kafka.log.LogManagerTest > testRecoveryDirectoryMappingWithRelativeDirectory 
PASSED

kafka.log.LogManagerTest > testGetNonExistentLog PASSED

kafka.log.LogManagerTest > testTwoLogManagersUsingSameDirFails PASSED

kafka.log.LogManagerTest > testLeastLoadedAssignment PASSED

kafka.log.LogManagerTest > testCleanupExpiredSegments PASSED

kafka.log.LogManagerTest > testCheckpointRecoveryPoints PASSED

kafka.log.LogManagerTest > testTimeBasedFlush PASSED

kafka.log.LogManagerTest > testCreateLog PASSED

kafka.log.LogManagerTest > testRecoveryDirectoryMappingWithTrailingSlash PASSED

kafka.coordinator.MemberMetadataTest > testMatchesSupportedProtocols PASSED

kafka.coordinator.MemberMetadataTest > testMetadata PASSED

kafka.coordinator.MemberMetadataTest > testMetadataRaisesOnUnsupportedProtocol 
PASSED

kafka.coordinator.MemberMetadataTest > testVoteForPreferredProtocol PASSED

kafka.coordinator.MemberMetadataTest > testVoteRaisesOnNoSupportedProtocols 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testHeartbeatWrongCoordinator 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testDescribeGroupStable PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testHeartbeatIllegalGeneration 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testDescribeGroupWrongCoordinator PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testDescribeGroupRebalancing 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testLeaderFailureInSyncGroup 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testGenerationIdIncrementsOnRebalance PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testSyncGroupFromIllegalGeneration PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testInvalidGroupId PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testHeartbeatUnknownGroup 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testListGroupsIncludesStableGroups PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testHeartbeatDuringRebalanceCausesRebalanceInProgress PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupInconsistentGroupProtocol PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupSessionTimeoutTooLarge PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupSessionTimeoutTooSmall PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testSyncGroupEmptyAssignment 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testCommitOffsetWithDefaultGeneration PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupFromUnchangedLeaderShouldRebalance PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testHeartbeatRebalanceInProgress PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testLeaveGroupUnknownGroup 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testListGroupsIncludesRebalancingGroups PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testSyncGroupFollowerAfterLeader PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testCommitOffsetInAwaitingSync 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testJoinGroupWrongCoordinator 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupUnknownConsumerExistingGroup PASSED

[jira] [Commented] (KAFKA-3205) Error in I/O with host (java.io.EOFException) raised in producer

2016-03-31 Thread Flavio Junqueira (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15220339#comment-15220339
 ] 

Flavio Junqueira commented on KAFKA-3205:
-

[~bondj] I've commented on the pull request.

> Error in I/O with host (java.io.EOFException) raised in producer
> 
>
> Key: KAFKA-3205
> URL: https://issues.apache.org/jira/browse/KAFKA-3205
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.8.2.1, 0.9.0.0
>Reporter: Jonathan Raffre
> Fix For: 0.10.0.0
>
>
> In a situation with a Kafka broker in 0.9 and producers still in 0.8.2.x, 
> producers seems to raise the following after a variable amount of time since 
> start :
> {noformat}
> 2016-01-29 14:33:13,066 WARN [] o.a.k.c.n.Selector: Error in I/O with 
> 172.22.2.170
> java.io.EOFException: null
> at 
> org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:62)
>  ~[org.apache.kafka.kafka-clients-0.8.2.0.jar:na]
> at org.apache.kafka.common.network.Selector.poll(Selector.java:248) 
> ~[org.apache.kafka.kafka-clients-0.8.2.0.jar:na]
> at 
> org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:192) 
> [org.apache.kafka.kafka-clients-0.8.2.0.jar:na]
> at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:191) 
> [org.apache.kafka.kafka-clients-0.8.2.0.jar:na]
> at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:122) 
> [org.apache.kafka.kafka-clients-0.8.2.0.jar:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66-internal]
> {noformat}
> This can be reproduced successfully by doing the following :
>  * Start a 0.8.2 producer connected to the 0.9 broker
>  * Wait 15 minutes, exactly
>  * See the error in the producer logs.
> Oddly, this also shows up in an active producer but after 10 minutes of 
> activity.
> Kafka's server.properties :
> {noformat}
> broker.id=1
> listeners=PLAINTEXT://:9092
> port=9092
> num.network.threads=2
> num.io.threads=2
> socket.send.buffer.bytes=1048576
> socket.receive.buffer.bytes=1048576
> socket.request.max.bytes=104857600
> log.dirs=/mnt/data/kafka
> num.partitions=4
> auto.create.topics.enable=false
> delete.topic.enable=true
> num.recovery.threads.per.data.dir=1
> log.retention.hours=48
> log.retention.bytes=524288000
> log.segment.bytes=52428800
> log.retention.check.interval.ms=6
> log.roll.hours=24
> log.cleanup.policy=delete
> log.cleaner.enable=true
> zookeeper.connect=127.0.0.1:2181
> zookeeper.connection.timeout.ms=100
> {noformat}
> Producer's configuration :
> {noformat}
>   compression.type = none
>   metric.reporters = []
>   metadata.max.age.ms = 30
>   metadata.fetch.timeout.ms = 6
>   acks = all
>   batch.size = 16384
>   reconnect.backoff.ms = 10
>   bootstrap.servers = [127.0.0.1:9092]
>   receive.buffer.bytes = 32768
>   retry.backoff.ms = 500
>   buffer.memory = 33554432
>   timeout.ms = 3
>   key.serializer = class 
> org.apache.kafka.common.serialization.StringSerializer
>   retries = 3
>   max.request.size = 500
>   block.on.buffer.full = true
>   value.serializer = class 
> org.apache.kafka.common.serialization.StringSerializer
>   metrics.sample.window.ms = 3
>   send.buffer.bytes = 131072
>   max.in.flight.requests.per.connection = 5
>   metrics.num.samples = 2
>   linger.ms = 0
>   client.id = 
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] 0.10.0.0 RC1

2016-03-31 Thread Gwen Shapira

Good question :)

Lets say we want:
KIP-43, KIP-35, KIP-4-metadata and KIP-33 to get in. How much time do we
need?

I think 2 or 3 weeks is reasonable? Am I being too optimistic again?

Gwen


On Thu, Mar 31, 2016 at 10:42 AM, Becket Qin  wrote:

> Sure, do we have a new intended release close date?
>
> On Thu, Mar 31, 2016 at 10:23 AM, Gwen Shapira  wrote:
>
> > I'd LOVE to have KIP-33 get it.
> >
> > Can you work with Jun to make sure the timing will work?
> >
> > On Thu, Mar 31, 2016 at 9:20 AM, Becket Qin 
> wrote:
> >
> > > Hi Gwen,
> > >
> > > KIP-47 is voted and passed, but not merged yet. It is actually
> depending
> > on
> > > KIP-33 which is in the voting process.
> > >
> > > I know we discussed on the KIP hangout that we will do KIP-33 post
> > 0.10.0.
> > > But since 0.10.0 is delayed, maybe we can include KIP-33 given it is a
> > > long-wanted fix by many users.
> > > And I agree with Grant that it would be nice to have KIP-35 in this
> > > release.
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > >
> > > On Wed, Mar 30, 2016 at 2:35 PM, Gwen Shapira 
> wrote:
> > >
> > > > I think we are in agreement here.
> > > > Note that KIP-47 is already voted and merged AFAIK, so it will be in
> > > 0.10.0
> > > > either way (actually, it will be nice if someone tries using this
> > feature
> > > > on a the RC...)
> > > >
> > > > Gwen
> > > >
> > > > On Wed, Mar 30, 2016 at 11:53 AM, Grant Henke 
> > > wrote:
> > > >
> > > > > Below are my thoughts on the release features. I don't feel too
> > > strongly
> > > > > about it, but I figured I would lay out my current perspective on
> the
> > > > 0.10
> > > > > release.
> > > > >
> > > > > When considering the optimal time to do this major release the
> > > > > considerations I am weighing are:
> > > > >
> > > > >- Compatibility
> > > > >   - Are more breaking changes almost ready? We want major
> > releases
> > > to
> > > > >   be infrequent. Having the breaking changes/pieces in will
> help
> > > 0.10
> > > > > last
> > > > >   longer.
> > > > >   - Breaking again in 3 months is not favorable, especially for
> > > > >   clients.
> > > > >- Features vs Risk
> > > > >   - Is it worth it for a user to upgrade? Given there is risk
> in
> > a
> > > > >   major upgrade, are there enough features to take that risk
> on?
> > > > >   - The breaking change we have introduced is the timestamp
> > changes
> > > > >   (KIP-31/KIP-32), but users can't use it yet (KIP-33) beyond a
> > > purge
> > > > >   improvement.
> > > > >   - We did add Streams but that could technically be added in a
> > > 0.9.1
> > > > >   release. (I think)
> > > > >
> > > > > For those reasons I would prefer to block on these changes for 0.10
> > > > >
> > > > >- KIP-4's Metadata changes
> > > > >- Breaking wire protocol change
> > > > >   - Also, fixes critical issues and adds rack fields from
> KIP-36
> > > > >   - Patch available, need consensus/vote on metadata protocol
> > > change
> > > > >- KIP-35: Retrieving protocol version
> > > > >- Though Kafka's internal usage has not been fully vetted, we
> all
> > > > agree
> > > > >   its useful to be able to ask for the protocol versions
> > > > >   - The Kafka client community really wants this
> > > > >   - It can reduce the impact/breakage of future releases
> > > > >- KIP-33: Add a time based log index
> > > > >   - This was a major motivation for KIP-31/32 and adding it
> could
> > > > help
> > > > >   vet those changes to ensure we don't need a breaking change
> > later
> > > > >   - Adding a feature flag to disable it broker side has been
> > > > discussed
> > > > >   to mitigate risk
> > > > >   - It doesn't need to block the release if we are confident it
> > > won't
> > > > >   need a breaking change
> > > > >
> > > > > It would also be nice to get these in if ready, but I don't think
> the
> > > > need
> > > > > to block the release:
> > > > >
> > > > >- KIP-43: Kafka SASL enhancements
> > > > >   - Scope has been reduced
> > > > >   - Seams it could be added quickly
> > > > >   - Not breaking addition, so doesn't need to block
> > > > >- KIP-47: Add timestamp-based log deletion policy
> > > > >- Also leverages KIP-32/33 further vetting its implementation
> > > > >   - Not breaking addition, so doesn't need to block
> > > > >
> > > > > That said, I still think we should push to get a release candidate
> > in a
> > > > > reasonable amount of time (a couple weeks?). Hopefully that is
> > > feasible.
> > > > >
> > > > > Thanks,
> > > > > Grant
> > > > >
> > > > >
> > > > > On Wed, Mar 30, 2016 at 12:58 PM, Rajini Sivaram <
> > > > > rajinisiva...@googlemail.com> wrote:
> > > > >
> > > > > > If there is time, we will be very keen on including KIP-43 in the
> > > > 0.10.0
> > > > > >

Re: [VOTE] 0.10.0.0 RC1

2016-03-31 Thread Becket Qin

Sure, do we have a new intended release close date?

On Thu, Mar 31, 2016 at 10:23 AM, Gwen Shapira  wrote:

> I'd LOVE to have KIP-33 get it.
>
> Can you work with Jun to make sure the timing will work?
>
> On Thu, Mar 31, 2016 at 9:20 AM, Becket Qin  wrote:
>
> > Hi Gwen,
> >
> > KIP-47 is voted and passed, but not merged yet. It is actually depending
> on
> > KIP-33 which is in the voting process.
> >
> > I know we discussed on the KIP hangout that we will do KIP-33 post
> 0.10.0.
> > But since 0.10.0 is delayed, maybe we can include KIP-33 given it is a
> > long-wanted fix by many users.
> > And I agree with Grant that it would be nice to have KIP-35 in this
> > release.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> > On Wed, Mar 30, 2016 at 2:35 PM, Gwen Shapira  wrote:
> >
> > > I think we are in agreement here.
> > > Note that KIP-47 is already voted and merged AFAIK, so it will be in
> > 0.10.0
> > > either way (actually, it will be nice if someone tries using this
> feature
> > > on a the RC...)
> > >
> > > Gwen
> > >
> > > On Wed, Mar 30, 2016 at 11:53 AM, Grant Henke 
> > wrote:
> > >
> > > > Below are my thoughts on the release features. I don't feel too
> > strongly
> > > > about it, but I figured I would lay out my current perspective on the
> > > 0.10
> > > > release.
> > > >
> > > > When considering the optimal time to do this major release the
> > > > considerations I am weighing are:
> > > >
> > > >- Compatibility
> > > >   - Are more breaking changes almost ready? We want major
> releases
> > to
> > > >   be infrequent. Having the breaking changes/pieces in will help
> > 0.10
> > > > last
> > > >   longer.
> > > >   - Breaking again in 3 months is not favorable, especially for
> > > >   clients.
> > > >- Features vs Risk
> > > >   - Is it worth it for a user to upgrade? Given there is risk in
> a
> > > >   major upgrade, are there enough features to take that risk on?
> > > >   - The breaking change we have introduced is the timestamp
> changes
> > > >   (KIP-31/KIP-32), but users can't use it yet (KIP-33) beyond a
> > purge
> > > >   improvement.
> > > >   - We did add Streams but that could technically be added in a
> > 0.9.1
> > > >   release. (I think)
> > > >
> > > > For those reasons I would prefer to block on these changes for 0.10
> > > >
> > > >- KIP-4's Metadata changes
> > > >- Breaking wire protocol change
> > > >   - Also, fixes critical issues and adds rack fields from KIP-36
> > > >   - Patch available, need consensus/vote on metadata protocol
> > change
> > > >- KIP-35: Retrieving protocol version
> > > >- Though Kafka's internal usage has not been fully vetted, we all
> > > agree
> > > >   its useful to be able to ask for the protocol versions
> > > >   - The Kafka client community really wants this
> > > >   - It can reduce the impact/breakage of future releases
> > > >- KIP-33: Add a time based log index
> > > >   - This was a major motivation for KIP-31/32 and adding it could
> > > help
> > > >   vet those changes to ensure we don't need a breaking change
> later
> > > >   - Adding a feature flag to disable it broker side has been
> > > discussed
> > > >   to mitigate risk
> > > >   - It doesn't need to block the release if we are confident it
> > won't
> > > >   need a breaking change
> > > >
> > > > It would also be nice to get these in if ready, but I don't think the
> > > need
> > > > to block the release:
> > > >
> > > >- KIP-43: Kafka SASL enhancements
> > > >   - Scope has been reduced
> > > >   - Seams it could be added quickly
> > > >   - Not breaking addition, so doesn't need to block
> > > >- KIP-47: Add timestamp-based log deletion policy
> > > >- Also leverages KIP-32/33 further vetting its implementation
> > > >   - Not breaking addition, so doesn't need to block
> > > >
> > > > That said, I still think we should push to get a release candidate
> in a
> > > > reasonable amount of time (a couple weeks?). Hopefully that is
> > feasible.
> > > >
> > > > Thanks,
> > > > Grant
> > > >
> > > >
> > > > On Wed, Mar 30, 2016 at 12:58 PM, Rajini Sivaram <
> > > > rajinisiva...@googlemail.com> wrote:
> > > >
> > > > > If there is time, we will be very keen on including KIP-43 in the
> > > 0.10.0
> > > > > release.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Rajini
> > > > >
> > > > > On Wed, Mar 30, 2016 at 6:37 PM, Ashish Singh  >
> > > > wrote:
> > > > >
> > > > > > If it is possible, I am also in favor of having some time to
> > include
> > > a
> > > > > few
> > > > > > more KIPs in 0.10.
> > > > > >
> > > > > > On Wed, Mar 30, 2016 at 10:26 AM, Jason Gustafson <
> > > ja...@confluent.io>
> > > > > > wrote:
> > > > > >
> > > > > > > I think it would be nice to get a resolution on KIP-35 before
> the
> > >

[jira] [Commented] (KAFKA-3334) First message on new topic not actually being sent, no exception thrown

2016-03-31 Thread Ashish K Singh (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15220289#comment-15220289
 ] 

Ashish K Singh commented on KAFKA-3334:
---

Thanks for the response [~salex89]. I will wait to get response from 
[~becket_qin] and [~ijuma] before making changes to PR.

> First message on new topic not actually being sent, no exception thrown
> ---
>
> Key: KAFKA-3334
> URL: https://issues.apache.org/jira/browse/KAFKA-3334
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.0
> Environment: Linux, Java
>Reporter: Aleksandar Stojadinovic
>Assignee: Ashish K Singh
> Fix For: 0.10.1.0
>
>
> Although I've seen this issue pop around the internet in a few forms, I'm not 
> sure it is yet properly fixed. 
> When publishing to a new topic, with auto create-enabled, the java client 
> (0.9.0) shows this WARN message in the log, and the message is not sent 
> obviously:
> org.apache.kafka.clients.NetworkClient - Error while fetching metadata with 
> correlation id 0 : {file.topic=LEADER_NOT_AVAILABLE}
> In the meantime I see in the console the message that a log for partition is 
> created. The next messages are patched through normally, but the first one is 
> never sent. No exception is ever thrown, either by calling get on the future, 
> or with the async usage, like everything is perfect.
> I notice when I leave my application blocked on the get call, in the 
> debugger, then the message may be processed, but with significant delay. This 
> is consistent with another issue I found for the python client. Also, if I 
> call partitionsFor previously, the topic is created and the message is sent. 
> But it seems silly to call it every time, just to mitigate this issue.
> {code}
> Future recordMetadataFuture = producer.send(new 
> ProducerRecord<>(topic, key, file));
> RecordMetadata recordMetadata = recordMetadataFuture.get(30, 
> TimeUnit.SECONDS);
> {code}
> I hope I'm clear enough.
> Related similar (but not same) issues:
> https://issues.apache.org/jira/browse/KAFKA-1124
> https://github.com/dpkp/kafka-python/issues/150
> http://stackoverflow.com/questions/35187933/how-to-resolve-leader-not-available-kafka-error-when-trying-to-consume



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3334) First message on new topic not actually being sent, no exception thrown

2016-03-31 Thread Aleksandar Stojadinovic (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15220277#comment-15220277
 ] 

Aleksandar Stojadinovic commented on KAFKA-3334:


I agree about overexposing the internals (at least in one document). A mention 
in FAQ, or even better, a chapter about auto-created topics should be more like 
it. I'm mentioning a separate chapter because I believe it will catch more 
attention. If someone is aiming to use the auto-crated feature, they will 
probably take a second and throw a look. Also, people Googling "Kafka publish 
autocreate" will have that one high in the results. 

To potentially discourage the auto-create feature, is it an option to disable 
it by default? And when a user searches for it, they can see a word of notice 
in the configuration?

I mean, the best thing would be to never have the issue of missing a message, 
but I guess that's easier said than done :) .

> First message on new topic not actually being sent, no exception thrown
> ---
>
> Key: KAFKA-3334
> URL: https://issues.apache.org/jira/browse/KAFKA-3334
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.0
> Environment: Linux, Java
>Reporter: Aleksandar Stojadinovic
>Assignee: Ashish K Singh
> Fix For: 0.10.1.0
>
>
> Although I've seen this issue pop around the internet in a few forms, I'm not 
> sure it is yet properly fixed. 
> When publishing to a new topic, with auto create-enabled, the java client 
> (0.9.0) shows this WARN message in the log, and the message is not sent 
> obviously:
> org.apache.kafka.clients.NetworkClient - Error while fetching metadata with 
> correlation id 0 : {file.topic=LEADER_NOT_AVAILABLE}
> In the meantime I see in the console the message that a log for partition is 
> created. The next messages are patched through normally, but the first one is 
> never sent. No exception is ever thrown, either by calling get on the future, 
> or with the async usage, like everything is perfect.
> I notice when I leave my application blocked on the get call, in the 
> debugger, then the message may be processed, but with significant delay. This 
> is consistent with another issue I found for the python client. Also, if I 
> call partitionsFor previously, the topic is created and the message is sent. 
> But it seems silly to call it every time, just to mitigate this issue.
> {code}
> Future recordMetadataFuture = producer.send(new 
> ProducerRecord<>(topic, key, file));
> RecordMetadata recordMetadata = recordMetadataFuture.get(30, 
> TimeUnit.SECONDS);
> {code}
> I hope I'm clear enough.
> Related similar (but not same) issues:
> https://issues.apache.org/jira/browse/KAFKA-1124
> https://github.com/dpkp/kafka-python/issues/150
> http://stackoverflow.com/questions/35187933/how-to-resolve-leader-not-available-kafka-error-when-trying-to-consume



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3488) commitAsync() fails if metadata update creates new SASL/SSL connection

2016-03-31 Thread Jason Gustafson (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15220272#comment-15220272
 ] 

Jason Gustafson commented on KAFKA-3488:


This has come up a few times in slightly different scenarios, so it's 
definitely worth fixing. A few options come to mind:

1. The current behavior is basically a consequence of the fact that 
NetworkClient only accepts one request send for each connection on each 
invocation of poll(). If it supported multiple request sends, then we wouldn't 
have to touch ConsumerNetworkClient. For the consumer, requests are usually 
tiny, so we ought to be able to send a bunch of them at once.
2. We can have ConsumerNetworkClient "try harder" to get the requests out. 
Instead of throwing on the first send failure, we can have it go back into 
poll(). This should clear the pending request send in NetworkClient so that 
another can be sent. This is actually how it was written initially, so you can 
look back at the history to see what that looked like. I don't recall whether 
there was a great reason to change it, but I think we were concerned about the 
overhead from multiple iterations of poll().
3. We can reschedule the async commit to be sent later. Because send failure 
remains a possibility even with the above fixes, this is really the only way to 
guarantee that a commit eventually will get sent. Unfortunately, it introduces 
the possibility of commit reordering, so we need an approach to deal with that 
(e.g. sequence numbers).

Among the these options, (2) probably has the least risk, but it might be 
worthwhile seeing what (1) would look like. There may be other options as well 
that I haven't thought of. 

> commitAsync() fails if metadata update creates new SASL/SSL connection
> --
>
> Key: KAFKA-3488
> URL: https://issues.apache.org/jira/browse/KAFKA-3488
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.9.0.1
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
> Fix For: 0.10.0.0
>
>
> Sasl/SslConsumerTest.testSimpleConsumption() fails intermittently with a 
> failure in {{commitAsync()}}. The exception stack trace shows:
> {quote}
> kafka.api.SaslPlaintextConsumerTest.testSimpleConsumption FAILED
> java.lang.AssertionError: expected:<1> but was:<0>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:834)
>   at org.junit.Assert.assertEquals(Assert.java:645)
>   at org.junit.Assert.assertEquals(Assert.java:631)
>   at 
> kafka.api.BaseConsumerTest.awaitCommitCallback(BaseConsumerTest.scala:340)
>   at 
> kafka.api.BaseConsumerTest.testSimpleConsumption(BaseConsumerTest.scala:85)
> {quote}
> I have recreated this with some additional trace. The tests run with a very 
> small metadata expiry interval, triggering metadata updates quite often. If a 
> metadata request immediately following a {{commitAsync()}} call creates a new 
> SSL/SASL connection, {{ConsumerNetworkClient.poll}} returns to process the 
> connection handshake packets. Since {{ConsumerNetworkClient.poll}} discards 
> all unsent packets before returning from poll, this can result in the failure 
> of the commit - the callback is invoked with {{SendFailedException}}.
> I understand that {{ConsumerNetworkClient.poll()}} discards unsent packets 
> rather than buffer them to keep the code simple. And perhaps it is ok to fail 
> {{commitAsync}} occasionally since the callback does indicate that the caller 
> should retry. But it feels like an unnecessary limitation that requires error 
> handling in client applications when there are no real failures and makes it 
> much harder to test reliably. As special handling to fix issues like 
> KAFKA-3412, KAFKA-2672 adds more complexity to the code anyway, and because 
> it is much harder to debug failures that affect only SSL/SASL, it may be 
> worth considering improving this behaviour.
> I will see if I can submit a PR for the specific issue I was seeing with the 
> impact of handshakes on {{commitAsync()}}, but I will be interested in views 
> on improving the logic in {{ConsumerNetworkClient}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] 0.10.0.0 RC1

2016-03-31 Thread Gwen Shapira

I'd LOVE to have KIP-33 get it.

Can you work with Jun to make sure the timing will work?

On Thu, Mar 31, 2016 at 9:20 AM, Becket Qin  wrote:

> Hi Gwen,
>
> KIP-47 is voted and passed, but not merged yet. It is actually depending on
> KIP-33 which is in the voting process.
>
> I know we discussed on the KIP hangout that we will do KIP-33 post 0.10.0.
> But since 0.10.0 is delayed, maybe we can include KIP-33 given it is a
> long-wanted fix by many users.
> And I agree with Grant that it would be nice to have KIP-35 in this
> release.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Wed, Mar 30, 2016 at 2:35 PM, Gwen Shapira  wrote:
>
> > I think we are in agreement here.
> > Note that KIP-47 is already voted and merged AFAIK, so it will be in
> 0.10.0
> > either way (actually, it will be nice if someone tries using this feature
> > on a the RC...)
> >
> > Gwen
> >
> > On Wed, Mar 30, 2016 at 11:53 AM, Grant Henke 
> wrote:
> >
> > > Below are my thoughts on the release features. I don't feel too
> strongly
> > > about it, but I figured I would lay out my current perspective on the
> > 0.10
> > > release.
> > >
> > > When considering the optimal time to do this major release the
> > > considerations I am weighing are:
> > >
> > >- Compatibility
> > >   - Are more breaking changes almost ready? We want major releases
> to
> > >   be infrequent. Having the breaking changes/pieces in will help
> 0.10
> > > last
> > >   longer.
> > >   - Breaking again in 3 months is not favorable, especially for
> > >   clients.
> > >- Features vs Risk
> > >   - Is it worth it for a user to upgrade? Given there is risk in a
> > >   major upgrade, are there enough features to take that risk on?
> > >   - The breaking change we have introduced is the timestamp changes
> > >   (KIP-31/KIP-32), but users can't use it yet (KIP-33) beyond a
> purge
> > >   improvement.
> > >   - We did add Streams but that could technically be added in a
> 0.9.1
> > >   release. (I think)
> > >
> > > For those reasons I would prefer to block on these changes for 0.10
> > >
> > >- KIP-4's Metadata changes
> > >- Breaking wire protocol change
> > >   - Also, fixes critical issues and adds rack fields from KIP-36
> > >   - Patch available, need consensus/vote on metadata protocol
> change
> > >- KIP-35: Retrieving protocol version
> > >- Though Kafka's internal usage has not been fully vetted, we all
> > agree
> > >   its useful to be able to ask for the protocol versions
> > >   - The Kafka client community really wants this
> > >   - It can reduce the impact/breakage of future releases
> > >- KIP-33: Add a time based log index
> > >   - This was a major motivation for KIP-31/32 and adding it could
> > help
> > >   vet those changes to ensure we don't need a breaking change later
> > >   - Adding a feature flag to disable it broker side has been
> > discussed
> > >   to mitigate risk
> > >   - It doesn't need to block the release if we are confident it
> won't
> > >   need a breaking change
> > >
> > > It would also be nice to get these in if ready, but I don't think the
> > need
> > > to block the release:
> > >
> > >- KIP-43: Kafka SASL enhancements
> > >   - Scope has been reduced
> > >   - Seams it could be added quickly
> > >   - Not breaking addition, so doesn't need to block
> > >- KIP-47: Add timestamp-based log deletion policy
> > >- Also leverages KIP-32/33 further vetting its implementation
> > >   - Not breaking addition, so doesn't need to block
> > >
> > > That said, I still think we should push to get a release candidate in a
> > > reasonable amount of time (a couple weeks?). Hopefully that is
> feasible.
> > >
> > > Thanks,
> > > Grant
> > >
> > >
> > > On Wed, Mar 30, 2016 at 12:58 PM, Rajini Sivaram <
> > > rajinisiva...@googlemail.com> wrote:
> > >
> > > > If there is time, we will be very keen on including KIP-43 in the
> > 0.10.0
> > > > release.
> > > >
> > > > Thanks,
> > > >
> > > > Rajini
> > > >
> > > > On Wed, Mar 30, 2016 at 6:37 PM, Ashish Singh 
> > > wrote:
> > > >
> > > > > If it is possible, I am also in favor of having some time to
> include
> > a
> > > > few
> > > > > more KIPs in 0.10.
> > > > >
> > > > > On Wed, Mar 30, 2016 at 10:26 AM, Jason Gustafson <
> > ja...@confluent.io>
> > > > > wrote:
> > > > >
> > > > > > I think it would be nice to get a resolution on KIP-35 before the
> > > > > release.
> > > > > > We were reluctant to push it through when the timeline was tight
> > > > (because
> > > > > > of unclear implications), but if we have more time to consider
> it,
> > it
> > > > > > definitely would feel better not to have the issue hanging.
> > > > > >
> > > > > > -Jason
> > > > > >
> > > > > > On Wed, Mar 30, 2016 at 10:17 AM, Gwen Shapira <
> g...@confluent.io>
> > > > > wrote:

[jira] [Updated] (KAFKA-3451) Add basic HTML coverage report generation to gradle

2016-03-31 Thread Gwen Shapira (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira updated KAFKA-3451:

   Resolution: Fixed
Fix Version/s: 0.10.0.0
   Status: Resolved  (was: Patch Available)

Issue resolved by pull request 1121
[https://github.com/apache/kafka/pull/1121]

> Add basic HTML coverage report generation to gradle
> ---
>
> Key: KAFKA-3451
> URL: https://issues.apache.org/jira/browse/KAFKA-3451
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Grant Henke
>Assignee: Grant Henke
> Fix For: 0.10.1.0, 0.10.0.0
>
> Attachments: Jacoco-html.zip, scoverage.zip
>
>
> Having some basic ability to report and view coverage is valuable and a good 
> start. This may not be perfect and enhancements should be tracked under the 
> KAFKA-1722 umbrella, but its a start. 
> This will use Jacoco to report on the java projects and Scoverage to report 
> on the Scala projects (core). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3451) Add basic HTML coverage report generation to gradle

2016-03-31 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15220230#comment-15220230
 ] 

ASF GitHub Bot commented on KAFKA-3451:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1121


> Add basic HTML coverage report generation to gradle
> ---
>
> Key: KAFKA-3451
> URL: https://issues.apache.org/jira/browse/KAFKA-3451
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Grant Henke
>Assignee: Grant Henke
> Fix For: 0.10.1.0, 0.10.0.0
>
> Attachments: Jacoco-html.zip, scoverage.zip
>
>
> Having some basic ability to report and view coverage is valuable and a good 
> start. This may not be perfect and enhancements should be tracked under the 
> KAFKA-1722 umbrella, but its a start. 
> This will use Jacoco to report on the java projects and Scoverage to report 
> on the Scala projects (core). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3451: Add basic HTML coverage report gen...

2016-03-31 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1121


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-3205) Error in I/O with host (java.io.EOFException) raised in producer

2016-03-31 Thread Ismael Juma (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-3205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3205:
---
Fix Version/s: 0.10.0.0

> Error in I/O with host (java.io.EOFException) raised in producer
> 
>
> Key: KAFKA-3205
> URL: https://issues.apache.org/jira/browse/KAFKA-3205
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.8.2.1, 0.9.0.0
>Reporter: Jonathan Raffre
> Fix For: 0.10.0.0
>
>
> In a situation with a Kafka broker in 0.9 and producers still in 0.8.2.x, 
> producers seems to raise the following after a variable amount of time since 
> start :
> {noformat}
> 2016-01-29 14:33:13,066 WARN [] o.a.k.c.n.Selector: Error in I/O with 
> 172.22.2.170
> java.io.EOFException: null
> at 
> org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:62)
>  ~[org.apache.kafka.kafka-clients-0.8.2.0.jar:na]
> at org.apache.kafka.common.network.Selector.poll(Selector.java:248) 
> ~[org.apache.kafka.kafka-clients-0.8.2.0.jar:na]
> at 
> org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:192) 
> [org.apache.kafka.kafka-clients-0.8.2.0.jar:na]
> at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:191) 
> [org.apache.kafka.kafka-clients-0.8.2.0.jar:na]
> at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:122) 
> [org.apache.kafka.kafka-clients-0.8.2.0.jar:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66-internal]
> {noformat}
> This can be reproduced successfully by doing the following :
>  * Start a 0.8.2 producer connected to the 0.9 broker
>  * Wait 15 minutes, exactly
>  * See the error in the producer logs.
> Oddly, this also shows up in an active producer but after 10 minutes of 
> activity.
> Kafka's server.properties :
> {noformat}
> broker.id=1
> listeners=PLAINTEXT://:9092
> port=9092
> num.network.threads=2
> num.io.threads=2
> socket.send.buffer.bytes=1048576
> socket.receive.buffer.bytes=1048576
> socket.request.max.bytes=104857600
> log.dirs=/mnt/data/kafka
> num.partitions=4
> auto.create.topics.enable=false
> delete.topic.enable=true
> num.recovery.threads.per.data.dir=1
> log.retention.hours=48
> log.retention.bytes=524288000
> log.segment.bytes=52428800
> log.retention.check.interval.ms=6
> log.roll.hours=24
> log.cleanup.policy=delete
> log.cleaner.enable=true
> zookeeper.connect=127.0.0.1:2181
> zookeeper.connection.timeout.ms=100
> {noformat}
> Producer's configuration :
> {noformat}
>   compression.type = none
>   metric.reporters = []
>   metadata.max.age.ms = 30
>   metadata.fetch.timeout.ms = 6
>   acks = all
>   batch.size = 16384
>   reconnect.backoff.ms = 10
>   bootstrap.servers = [127.0.0.1:9092]
>   receive.buffer.bytes = 32768
>   retry.backoff.ms = 500
>   buffer.memory = 33554432
>   timeout.ms = 3
>   key.serializer = class 
> org.apache.kafka.common.serialization.StringSerializer
>   retries = 3
>   max.request.size = 500
>   block.on.buffer.full = true
>   value.serializer = class 
> org.apache.kafka.common.serialization.StringSerializer
>   metrics.sample.window.ms = 3
>   send.buffer.bytes = 131072
>   max.in.flight.requests.per.connection = 5
>   metrics.num.samples = 2
>   linger.ms = 0
>   client.id = 
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[DISCUSS] KIP-52 - Add Connector Control APIs

2016-03-31 Thread Jason Gustafson

Hi All, I've written a short KIP to add control APIs to Kafka Connect to
make administration easier:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-52%3A+Connector+Control+APIs.
Please let me know your thoughts.

Thanks,
Jason

Re: [VOTE] 0.10.0.0 RC1

2016-03-31 Thread Becket Qin

Hi Gwen,

KIP-47 is voted and passed, but not merged yet. It is actually depending on
KIP-33 which is in the voting process.

I know we discussed on the KIP hangout that we will do KIP-33 post 0.10.0.
But since 0.10.0 is delayed, maybe we can include KIP-33 given it is a
long-wanted fix by many users.
And I agree with Grant that it would be nice to have KIP-35 in this release.

Thanks,

Jiangjie (Becket) Qin

On Wed, Mar 30, 2016 at 2:35 PM, Gwen Shapira  wrote:

> I think we are in agreement here.
> Note that KIP-47 is already voted and merged AFAIK, so it will be in 0.10.0
> either way (actually, it will be nice if someone tries using this feature
> on a the RC...)
>
> Gwen
>
> On Wed, Mar 30, 2016 at 11:53 AM, Grant Henke  wrote:
>
> > Below are my thoughts on the release features. I don't feel too strongly
> > about it, but I figured I would lay out my current perspective on the
> 0.10
> > release.
> >
> > When considering the optimal time to do this major release the
> > considerations I am weighing are:
> >
> >- Compatibility
> >   - Are more breaking changes almost ready? We want major releases to
> >   be infrequent. Having the breaking changes/pieces in will help 0.10
> > last
> >   longer.
> >   - Breaking again in 3 months is not favorable, especially for
> >   clients.
> >- Features vs Risk
> >   - Is it worth it for a user to upgrade? Given there is risk in a
> >   major upgrade, are there enough features to take that risk on?
> >   - The breaking change we have introduced is the timestamp changes
> >   (KIP-31/KIP-32), but users can't use it yet (KIP-33) beyond a purge
> >   improvement.
> >   - We did add Streams but that could technically be added in a 0.9.1
> >   release. (I think)
> >
> > For those reasons I would prefer to block on these changes for 0.10
> >
> >- KIP-4's Metadata changes
> >- Breaking wire protocol change
> >   - Also, fixes critical issues and adds rack fields from KIP-36
> >   - Patch available, need consensus/vote on metadata protocol change
> >- KIP-35: Retrieving protocol version
> >- Though Kafka's internal usage has not been fully vetted, we all
> agree
> >   its useful to be able to ask for the protocol versions
> >   - The Kafka client community really wants this
> >   - It can reduce the impact/breakage of future releases
> >- KIP-33: Add a time based log index
> >   - This was a major motivation for KIP-31/32 and adding it could
> help
> >   vet those changes to ensure we don't need a breaking change later
> >   - Adding a feature flag to disable it broker side has been
> discussed
> >   to mitigate risk
> >   - It doesn't need to block the release if we are confident it won't
> >   need a breaking change
> >
> > It would also be nice to get these in if ready, but I don't think the
> need
> > to block the release:
> >
> >- KIP-43: Kafka SASL enhancements
> >   - Scope has been reduced
> >   - Seams it could be added quickly
> >   - Not breaking addition, so doesn't need to block
> >- KIP-47: Add timestamp-based log deletion policy
> >- Also leverages KIP-32/33 further vetting its implementation
> >   - Not breaking addition, so doesn't need to block
> >
> > That said, I still think we should push to get a release candidate in a
> > reasonable amount of time (a couple weeks?). Hopefully that is feasible.
> >
> > Thanks,
> > Grant
> >
> >
> > On Wed, Mar 30, 2016 at 12:58 PM, Rajini Sivaram <
> > rajinisiva...@googlemail.com> wrote:
> >
> > > If there is time, we will be very keen on including KIP-43 in the
> 0.10.0
> > > release.
> > >
> > > Thanks,
> > >
> > > Rajini
> > >
> > > On Wed, Mar 30, 2016 at 6:37 PM, Ashish Singh 
> > wrote:
> > >
> > > > If it is possible, I am also in favor of having some time to include
> a
> > > few
> > > > more KIPs in 0.10.
> > > >
> > > > On Wed, Mar 30, 2016 at 10:26 AM, Jason Gustafson <
> ja...@confluent.io>
> > > > wrote:
> > > >
> > > > > I think it would be nice to get a resolution on KIP-35 before the
> > > > release.
> > > > > We were reluctant to push it through when the timeline was tight
> > > (because
> > > > > of unclear implications), but if we have more time to consider it,
> it
> > > > > definitely would feel better not to have the issue hanging.
> > > > >
> > > > > -Jason
> > > > >
> > > > > On Wed, Mar 30, 2016 at 10:17 AM, Gwen Shapira 
> > > > wrote:
> > > > >
> > > > > > 1. Thank you, Guozhang. We'll roll out a new RC early next week
> if
> > > this
> > > > > is
> > > > > > ok?
> > > > > >
> > > > > > 2. I'm seeing very little community feedback (positive or
> negative)
> > > on
> > > > > the
> > > > > > release vote.
> > > > > > Would people feel better about the release if we delay it a bit
> to
> > > get
> > > > > some
> > > > > > high-priority and really awesome KIPs in?

[jira] [Updated] (KAFKA-2910) Failure in kafka.api.SslEndToEndAuthorizationTest.testNoGroupAcl

2016-03-31 Thread Rajini Sivaram (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-2910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajini Sivaram updated KAFKA-2910:
--
Fix Version/s: (was: 0.10.1.0)
   0.10.0.0

> Failure in kafka.api.SslEndToEndAuthorizationTest.testNoGroupAcl
> 
>
> Key: KAFKA-2910
> URL: https://issues.apache.org/jira/browse/KAFKA-2910
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Guozhang Wang
>Assignee: Rajini Sivaram
> Fix For: 0.10.0.0
>
>
> {code}
> java.lang.SecurityException: zkEnableSecureAcls is true, but the verification 
> of the JAAS login file failed.
>   at kafka.server.KafkaServer.initZk(KafkaServer.scala:265)
>   at kafka.server.KafkaServer.startup(KafkaServer.scala:168)
>   at kafka.utils.TestUtils$.createServer(TestUtils.scala:143)
>   at 
> kafka.integration.KafkaServerTestHarness$$anonfun$setUp$1.apply(KafkaServerTestHarness.scala:66)
>   at 
> kafka.integration.KafkaServerTestHarness$$anonfun$setUp$1.apply(KafkaServerTestHarness.scala:66)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
>   at scala.collection.Iterator$class.foreach(Iterator.scala:742)
>   at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
>   at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
>   at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
>   at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
>   at scala.collection.AbstractTraversable.map(Traversable.scala:104)
>   at 
> kafka.integration.KafkaServerTestHarness$class.setUp(KafkaServerTestHarness.scala:66)
>   at 
> kafka.api.SslEndToEndAuthorizationTest.kafka$api$IntegrationTestHarness$$super$setUp(SslEndToEndAuthorizationTest.scala:24)
>   at 
> kafka.api.IntegrationTestHarness$class.setUp(IntegrationTestHarness.scala:58)
>   at 
> kafka.api.SslEndToEndAuthorizationTest.kafka$api$EndToEndAuthorizationTest$$super$setUp(SslEndToEndAuthorizationTest.scala:24)
>   at 
> kafka.api.EndToEndAuthorizationTest$class.setUp(EndToEndAuthorizationTest.scala:141)
>   at 
> kafka.api.SslEndToEndAuthorizationTest.setUp(SslEndToEndAuthorizationTest.scala:24)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:105)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:56)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:64)
>   at 
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:50)
>   at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
>   at 
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
>   at 
>

[jira] [Commented] (KAFKA-2910) Failure in kafka.api.SslEndToEndAuthorizationTest.testNoGroupAcl

2016-03-31 Thread Rajini Sivaram (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-2910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15219967#comment-15219967
 ] 

Rajini Sivaram commented on KAFKA-2910:
---

[~ijuma] Yes, I will try out a fix, do some testing and then submit a PR. Have 
set the fix version to 0.10.0.0. Thanks.

> Failure in kafka.api.SslEndToEndAuthorizationTest.testNoGroupAcl
> 
>
> Key: KAFKA-2910
> URL: https://issues.apache.org/jira/browse/KAFKA-2910
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Guozhang Wang
>Assignee: Rajini Sivaram
> Fix For: 0.10.0.0
>
>
> {code}
> java.lang.SecurityException: zkEnableSecureAcls is true, but the verification 
> of the JAAS login file failed.
>   at kafka.server.KafkaServer.initZk(KafkaServer.scala:265)
>   at kafka.server.KafkaServer.startup(KafkaServer.scala:168)
>   at kafka.utils.TestUtils$.createServer(TestUtils.scala:143)
>   at 
> kafka.integration.KafkaServerTestHarness$$anonfun$setUp$1.apply(KafkaServerTestHarness.scala:66)
>   at 
> kafka.integration.KafkaServerTestHarness$$anonfun$setUp$1.apply(KafkaServerTestHarness.scala:66)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
>   at scala.collection.Iterator$class.foreach(Iterator.scala:742)
>   at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
>   at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
>   at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
>   at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
>   at scala.collection.AbstractTraversable.map(Traversable.scala:104)
>   at 
> kafka.integration.KafkaServerTestHarness$class.setUp(KafkaServerTestHarness.scala:66)
>   at 
> kafka.api.SslEndToEndAuthorizationTest.kafka$api$IntegrationTestHarness$$super$setUp(SslEndToEndAuthorizationTest.scala:24)
>   at 
> kafka.api.IntegrationTestHarness$class.setUp(IntegrationTestHarness.scala:58)
>   at 
> kafka.api.SslEndToEndAuthorizationTest.kafka$api$EndToEndAuthorizationTest$$super$setUp(SslEndToEndAuthorizationTest.scala:24)
>   at 
> kafka.api.EndToEndAuthorizationTest$class.setUp(EndToEndAuthorizationTest.scala:141)
>   at 
> kafka.api.SslEndToEndAuthorizationTest.setUp(SslEndToEndAuthorizationTest.scala:24)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:105)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:56)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:64)
>   at 
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:50)
>   at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
>   at 
>

[jira] [Commented] (KAFKA-2910) Failure in kafka.api.SslEndToEndAuthorizationTest.testNoGroupAcl

2016-03-31 Thread Ismael Juma (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-2910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15219956#comment-15219956
 ] 

Ismael Juma commented on KAFKA-2910:


Interesting, I think we have at least one other JIRA with the same root cause 
as Flavio said (I'll try to find it later). Will you be able to provide a PR 
for this? If so, please set the fix version to 0.10.0.0 as we are very keen on 
improving test stability.

> Failure in kafka.api.SslEndToEndAuthorizationTest.testNoGroupAcl
> 
>
> Key: KAFKA-2910
> URL: https://issues.apache.org/jira/browse/KAFKA-2910
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Guozhang Wang
>Assignee: Rajini Sivaram
> Fix For: 0.10.1.0
>
>
> {code}
> java.lang.SecurityException: zkEnableSecureAcls is true, but the verification 
> of the JAAS login file failed.
>   at kafka.server.KafkaServer.initZk(KafkaServer.scala:265)
>   at kafka.server.KafkaServer.startup(KafkaServer.scala:168)
>   at kafka.utils.TestUtils$.createServer(TestUtils.scala:143)
>   at 
> kafka.integration.KafkaServerTestHarness$$anonfun$setUp$1.apply(KafkaServerTestHarness.scala:66)
>   at 
> kafka.integration.KafkaServerTestHarness$$anonfun$setUp$1.apply(KafkaServerTestHarness.scala:66)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
>   at scala.collection.Iterator$class.foreach(Iterator.scala:742)
>   at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
>   at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
>   at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
>   at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
>   at scala.collection.AbstractTraversable.map(Traversable.scala:104)
>   at 
> kafka.integration.KafkaServerTestHarness$class.setUp(KafkaServerTestHarness.scala:66)
>   at 
> kafka.api.SslEndToEndAuthorizationTest.kafka$api$IntegrationTestHarness$$super$setUp(SslEndToEndAuthorizationTest.scala:24)
>   at 
> kafka.api.IntegrationTestHarness$class.setUp(IntegrationTestHarness.scala:58)
>   at 
> kafka.api.SslEndToEndAuthorizationTest.kafka$api$EndToEndAuthorizationTest$$super$setUp(SslEndToEndAuthorizationTest.scala:24)
>   at 
> kafka.api.EndToEndAuthorizationTest$class.setUp(EndToEndAuthorizationTest.scala:141)
>   at 
> kafka.api.SslEndToEndAuthorizationTest.setUp(SslEndToEndAuthorizationTest.scala:24)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:105)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:56)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:64)
>   at 
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:50)
>   at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at

[jira] [Commented] (KAFKA-2910) Failure in kafka.api.SslEndToEndAuthorizationTest.testNoGroupAcl

2016-03-31 Thread Rajini Sivaram (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-2910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15219926#comment-15219926
 ] 

Rajini Sivaram commented on KAFKA-2910:
---

Came across this failure while running the tests locally. I think the issue is 
with tests which don't close Zookeeper clients. Unclosed clients continue to 
attempt to reconnect to the Zookeeper server after the server is shutdown, and 
this loads JAAS configuration. Subsequent tests reuse the JAAS configuration, 
resulting in this transient failure.

One possible sequence causing failure:
# TestA creates Zookeeper server in its setUp
# TestA creates ZkUtils in a test
# TestA shuts down Zookeeper server in its tearDown
# TestA calls {{Configuration.setConfiguration(null)}} in its tearDown to reset 
JAAS config
# TestB starts
# TestA's ZK client sender thread which was not closed attempts to reconnect, 
calling {{Configuration.getConfiguration()}}. JAAS config is now reloaded 
because it was reset by TestA's tearDown. At this point the JAAS config loaded 
is typically an empty config.
# TestB creates JAAS config file and sets System property 
{{java.security.auth.login.config}}
# TestB creates Zookeeper and Kafka servers, expecting 
{{Configuration.getConfiguration()}} to load the config based on the currently 
set System property {{java.security.auth.login.config}}. But since the 
configuration was already loaded in  step 6) by TestA before the System 
property was set, JAAS config is not reloaded. TestB setUp fails as a result.

> Failure in kafka.api.SslEndToEndAuthorizationTest.testNoGroupAcl
> 
>
> Key: KAFKA-2910
> URL: https://issues.apache.org/jira/browse/KAFKA-2910
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Guozhang Wang
>Assignee: Rajini Sivaram
> Fix For: 0.10.1.0
>
>
> {code}
> java.lang.SecurityException: zkEnableSecureAcls is true, but the verification 
> of the JAAS login file failed.
>   at kafka.server.KafkaServer.initZk(KafkaServer.scala:265)
>   at kafka.server.KafkaServer.startup(KafkaServer.scala:168)
>   at kafka.utils.TestUtils$.createServer(TestUtils.scala:143)
>   at 
> kafka.integration.KafkaServerTestHarness$$anonfun$setUp$1.apply(KafkaServerTestHarness.scala:66)
>   at 
> kafka.integration.KafkaServerTestHarness$$anonfun$setUp$1.apply(KafkaServerTestHarness.scala:66)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
>   at scala.collection.Iterator$class.foreach(Iterator.scala:742)
>   at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
>   at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
>   at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
>   at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
>   at scala.collection.AbstractTraversable.map(Traversable.scala:104)
>   at 
> kafka.integration.KafkaServerTestHarness$class.setUp(KafkaServerTestHarness.scala:66)
>   at 
> kafka.api.SslEndToEndAuthorizationTest.kafka$api$IntegrationTestHarness$$super$setUp(SslEndToEndAuthorizationTest.scala:24)
>   at 
> kafka.api.IntegrationTestHarness$class.setUp(IntegrationTestHarness.scala:58)
>   at 
> kafka.api.SslEndToEndAuthorizationTest.kafka$api$EndToEndAuthorizationTest$$super$setUp(SslEndToEndAuthorizationTest.scala:24)
>   at 
> kafka.api.EndToEndAuthorizationTest$class.setUp(EndToEndAuthorizationTest.scala:141)
>   at 
> kafka.api.SslEndToEndAuthorizationTest.setUp(SslEndToEndAuthorizationTest.scala:24)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>

Re: KIP-4 Wiki Update

2016-03-31 Thread Grant Henke

>
> I had a second look at the proposed changes to Metadata Request and
> Response and it seems to me that having a `controller_id` field would be
> more efficient for non-trivial cases than having a `is_controller` field
>  for each broker (which would be false for all but 1 case).


I agree this is better. I will update it.

Similar, but less clear is the best way to encode `marked_for_deletion` and
> `is_internal`. These will also be false for most topics (there is only one
> internal topic at the moment, for example), so it may make sense to have a
> topics_marked_for_deletion and internal_topics in the response. Because
> topics are identified by strings, it is not as clear-cut as the
> controller_id case, but it still seems like it would be a win for when it
> matters most (when the number of topics is large).
>

Thats an interesting idea. I can try making this change to see what it
would look like.

Thanks,
Grant

On Thu, Mar 31, 2016 at 8:59 AM, Ismael Juma  wrote:

> Hi Grant,
>
> I had a second look at the proposed changes to Metadata Request and
> Response and it seems to me that having a `controller_id` field would be
> more efficient for non-trivial cases than having a `is_controller` field
>  for each broker (which would be false for all but 1 case).
>
> Similar, but less clear is the best way to encode `marked_for_deletion` and
> `is_internal`. These will also be false for most topics (there is only one
> internal topic at the moment, for example), so it may make sense to have a
> topics_marked_for_deletion and internal_topics in the response. Because
> topics are identified by strings, it is not as clear-cut as the
> controller_id case, but it still seems like it would be a win for when it
> matters most (when the number of topics is large).


> Ismael
>
> On Mon, Mar 14, 2016 at 10:07 PM, Grant Henke  wrote:
>
> > I have been updating the KIP-4 wiki page based on the last KIP call and
> > wanted to get some review and discussion around the server side
> > implementation for admin requests. Both the "ideal" functionality and the
> > "intermediated" functionality. The updates are still in progress, but
> this
> > section is the most critical and will likely have the most discussion.
> This
> > topic has had a few shifts in perspective and various discussions on
> > synchronous vs asynchronous server support. The wiki contains my current
> > perspective on the challenges and approach.
> >
> > If you have any thoughts or feedback on the "Server-side Admin Request
> > handlers" section here
> > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-2.Server-sideAdminRequesthandlers
> > >.
> > Lets discuss them in this thread.
> >
> > For reference the last KIP discussion can be viewed here:
> > https://youtu.be/rFW0-zJqg5I?t=12m30s
> >
> > Thank you,
> > Grant
> > --
> > Grant Henke
> > Software Engineer | Cloudera
> > gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
> >
>



-- 
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke

[jira] [Assigned] (KAFKA-2910) Failure in kafka.api.SslEndToEndAuthorizationTest.testNoGroupAcl

2016-03-31 Thread Rajini Sivaram (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-2910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajini Sivaram reassigned KAFKA-2910:
-

Assignee: Rajini Sivaram

> Failure in kafka.api.SslEndToEndAuthorizationTest.testNoGroupAcl
> 
>
> Key: KAFKA-2910
> URL: https://issues.apache.org/jira/browse/KAFKA-2910
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Guozhang Wang
>Assignee: Rajini Sivaram
> Fix For: 0.10.1.0
>
>
> {code}
> java.lang.SecurityException: zkEnableSecureAcls is true, but the verification 
> of the JAAS login file failed.
>   at kafka.server.KafkaServer.initZk(KafkaServer.scala:265)
>   at kafka.server.KafkaServer.startup(KafkaServer.scala:168)
>   at kafka.utils.TestUtils$.createServer(TestUtils.scala:143)
>   at 
> kafka.integration.KafkaServerTestHarness$$anonfun$setUp$1.apply(KafkaServerTestHarness.scala:66)
>   at 
> kafka.integration.KafkaServerTestHarness$$anonfun$setUp$1.apply(KafkaServerTestHarness.scala:66)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
>   at scala.collection.Iterator$class.foreach(Iterator.scala:742)
>   at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
>   at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
>   at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
>   at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
>   at scala.collection.AbstractTraversable.map(Traversable.scala:104)
>   at 
> kafka.integration.KafkaServerTestHarness$class.setUp(KafkaServerTestHarness.scala:66)
>   at 
> kafka.api.SslEndToEndAuthorizationTest.kafka$api$IntegrationTestHarness$$super$setUp(SslEndToEndAuthorizationTest.scala:24)
>   at 
> kafka.api.IntegrationTestHarness$class.setUp(IntegrationTestHarness.scala:58)
>   at 
> kafka.api.SslEndToEndAuthorizationTest.kafka$api$EndToEndAuthorizationTest$$super$setUp(SslEndToEndAuthorizationTest.scala:24)
>   at 
> kafka.api.EndToEndAuthorizationTest$class.setUp(EndToEndAuthorizationTest.scala:141)
>   at 
> kafka.api.SslEndToEndAuthorizationTest.setUp(SslEndToEndAuthorizationTest.scala:24)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:105)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:56)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:64)
>   at 
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:50)
>   at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
>   at 
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
>   at 
>

[jira] [Commented] (KAFKA-3456) In-house KafkaMetric misreports metrics when periodically observed

2016-03-31 Thread The Data Lorax (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15219894#comment-15219894
 ] 

The Data Lorax commented on KAFKA-3456:
---

OK, I'm getting the feeling there is little appetite to change the way this 
works and people would rather stick with the known limitations of the metrics.  
That being the case I'm happy(ish) for this to be closed.  If on the other hand 
we'd prefer more correct metrics, then I'm happy to work on a PR...

> In-house KafkaMetric misreports metrics when periodically observed
> --
>
> Key: KAFKA-3456
> URL: https://issues.apache.org/jira/browse/KAFKA-3456
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer, core, producer 
>Affects Versions: 0.9.0.0, 0.9.0.1, 0.10.0.0
>Reporter: The Data Lorax
>Assignee: Neha Narkhede
>Priority: Minor
>
> The metrics captured by Kafka through the in-house {{SampledStat}} suffer 
> from misreporting metrics if observed in a periodic manner.
> Consider a {{Rate}} metric that is using the default 2 samples and 30 second 
> sample window i.e. the {{Rate}} is capturing 60 seconds worth of data.  So, 
> to report this metric to some external system we might poll it every 60 
> seconds to observe the current value. Using a shorter period would, in the 
> case of a {{Rate}}, lead to smoothing of the plotted data, and worse, in the 
> case of a {{Count}}, would lead to double counting - so 60 seconds is the 
> only period at which we can poll the metrics if we are to report accurate 
> metrics.
> To demonstrate the issue consider the following somewhat extreme case:
> The {{Rate}}  is capturing data from a system which alternates between a 999 
> per sec rate and a 1 per sec rate every 30 seconds, with the different rates 
> aligned with the sample boundaries within the {{Rate}} instance i.e. after 60 
> seconds the first sample within the {{Rate}} instance will have a rate of 999 
> per sec, and the second 1 per sec. 
> If we were to ask the metric for its value at this 60 second boundary it 
> would correctly report 500 per sec. However, if we asked it again 1 
> millisecond later it would report 1 per sec, as the first sample window has 
> been aged out. Depending on how retarded into the 60 sec period of the metric 
> our periodic poll of the metric was, we would observe a constant rate 
> somewhere in the range of 1 to 500 per second, most likely around the 250 
> mark. 
> Other metrics based off of the {{SampledStat}} type suffer from the same 
> issue e.g. the {{Count}} metric, given a constant rate of 1 per second, will 
> report a constant count somewhere between 30 and 60, rather than the correct 
> 60.
> This can be seen in the following test code:
> {code:java}
> public class MetricsTest {
> private MetricConfig metricsConfig;
> @Before
> public void setUp() throws Exception {
> metricsConfig = new MetricConfig();
> }
> private long t(final int bucket) {
> return metricsConfig.timeWindowMs() * bucket;
> }
> @Test
> public void testHowRateDropsMetrics() throws Exception {
> Rate rate = new Rate();
> metricsConfig.samples(2);
> metricsConfig.timeWindow(30, TimeUnit.SECONDS);
> // First sample window from t0 -> (t1 -1), with rate 999 per second:
> for (long time = t(0); time != t(1); time += 1000) {
> rate.record(metricsConfig, 999, time);
> }
> // Second sample window from t1 -> (t2 -1), with rate 1 per second:
> for (long time = t(1); time != t(2); time += 1000) {
> rate.record(metricsConfig, 1, time);
> }
> // Measure at bucket boundary, (though same issue exists all periodic 
> measurements)
> final double m1 = rate.measure(metricsConfig, t(2));// m1 = 1.0
> // Third sample window from t2 -> (t3 -1), with rate 999 per second:
> for (long time = t(2); time != t(3); time += 1000) {
> rate.record(metricsConfig, 999, time);
> }
> // Second sample window from t3 -> (t4 -1), with rate 1 per second:
> for (long time = t(3); time != t(4); time += 1000) {
> rate.record(metricsConfig, 1, time);
> }
> // Measure second pair of samples:
> final double m2 = rate.measure(metricsConfig, t(4));// m2 = 1.0
> assertEquals("Measurement of the rate over the first two samples", 
> 500.0, m1, 2.0);
> assertEquals("Measurement of the rate over the last two samples", 
> 500.0, m2, 2.0);
> }
> @Test
> public void testHowRateDropsMetricsWithRetardedObservations() throws 
> Exception {
> final long retardation = 1000;
> Rate rate = new Rate();
> metricsConfig.samples(2);
>

Re: KIP-4 Wiki Update

2016-03-31 Thread Ismael Juma

Hi Grant,

I had a second look at the proposed changes to Metadata Request and
Response and it seems to me that having a `controller_id` field would be
more efficient for non-trivial cases than having a `is_controller` field
 for each broker (which would be false for all but 1 case).

Similar, but less clear is the best way to encode `marked_for_deletion` and
`is_internal`. These will also be false for most topics (there is only one
internal topic at the moment, for example), so it may make sense to have a
topics_marked_for_deletion and internal_topics in the response. Because
topics are identified by strings, it is not as clear-cut as the
controller_id case, but it still seems like it would be a win for when it
matters most (when the number of topics is large).

Ismael

On Mon, Mar 14, 2016 at 10:07 PM, Grant Henke  wrote:

> I have been updating the KIP-4 wiki page based on the last KIP call and
> wanted to get some review and discussion around the server side
> implementation for admin requests. Both the "ideal" functionality and the
> "intermediated" functionality. The updates are still in progress, but this
> section is the most critical and will likely have the most discussion. This
> topic has had a few shifts in perspective and various discussions on
> synchronous vs asynchronous server support. The wiki contains my current
> perspective on the challenges and approach.
>
> If you have any thoughts or feedback on the "Server-side Admin Request
> handlers" section here
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-2.Server-sideAdminRequesthandlers
> >.
> Lets discuss them in this thread.
>
> For reference the last KIP discussion can be viewed here:
> https://youtu.be/rFW0-zJqg5I?t=12m30s
>
> Thank you,
> Grant
> --
> Grant Henke
> Software Engineer | Cloudera
> gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
>

[jira] [Updated] (KAFKA-3488) commitAsync() fails if metadata update creates new SASL/SSL connection

2016-03-31 Thread Ismael Juma (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-3488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3488:
---
Fix Version/s: 0.10.0.0

> commitAsync() fails if metadata update creates new SASL/SSL connection
> --
>
> Key: KAFKA-3488
> URL: https://issues.apache.org/jira/browse/KAFKA-3488
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.9.0.1
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
> Fix For: 0.10.0.0
>
>
> Sasl/SslConsumerTest.testSimpleConsumption() fails intermittently with a 
> failure in {{commitAsync()}}. The exception stack trace shows:
> {quote}
> kafka.api.SaslPlaintextConsumerTest.testSimpleConsumption FAILED
> java.lang.AssertionError: expected:<1> but was:<0>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:834)
>   at org.junit.Assert.assertEquals(Assert.java:645)
>   at org.junit.Assert.assertEquals(Assert.java:631)
>   at 
> kafka.api.BaseConsumerTest.awaitCommitCallback(BaseConsumerTest.scala:340)
>   at 
> kafka.api.BaseConsumerTest.testSimpleConsumption(BaseConsumerTest.scala:85)
> {quote}
> I have recreated this with some additional trace. The tests run with a very 
> small metadata expiry interval, triggering metadata updates quite often. If a 
> metadata request immediately following a {{commitAsync()}} call creates a new 
> SSL/SASL connection, {{ConsumerNetworkClient.poll}} returns to process the 
> connection handshake packets. Since {{ConsumerNetworkClient.poll}} discards 
> all unsent packets before returning from poll, this can result in the failure 
> of the commit - the callback is invoked with {{SendFailedException}}.
> I understand that {{ConsumerNetworkClient.poll()}} discards unsent packets 
> rather than buffer them to keep the code simple. And perhaps it is ok to fail 
> {{commitAsync}} occasionally since the callback does indicate that the caller 
> should retry. But it feels like an unnecessary limitation that requires error 
> handling in client applications when there are no real failures and makes it 
> much harder to test reliably. As special handling to fix issues like 
> KAFKA-3412, KAFKA-2672 adds more complexity to the code anyway, and because 
> it is much harder to debug failures that affect only SSL/SASL, it may be 
> worth considering improving this behaviour.
> I will see if I can submit a PR for the specific issue I was seeing with the 
> impact of handshakes on {{commitAsync()}}, but I will be interested in views 
> on improving the logic in {{ConsumerNetworkClient}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3488) commitAsync() fails if metadata update creates new SASL/SSL connection

2016-03-31 Thread Ismael Juma (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15219731#comment-15219731
 ] 

Ismael Juma commented on KAFKA-3488:


Good catch [~rsivaram]. Thoughts [~hachikuji]?

> commitAsync() fails if metadata update creates new SASL/SSL connection
> --
>
> Key: KAFKA-3488
> URL: https://issues.apache.org/jira/browse/KAFKA-3488
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.9.0.1
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
>
> Sasl/SslConsumerTest.testSimpleConsumption() fails intermittently with a 
> failure in {{commitAsync()}}. The exception stack trace shows:
> {quote}
> kafka.api.SaslPlaintextConsumerTest.testSimpleConsumption FAILED
> java.lang.AssertionError: expected:<1> but was:<0>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:834)
>   at org.junit.Assert.assertEquals(Assert.java:645)
>   at org.junit.Assert.assertEquals(Assert.java:631)
>   at 
> kafka.api.BaseConsumerTest.awaitCommitCallback(BaseConsumerTest.scala:340)
>   at 
> kafka.api.BaseConsumerTest.testSimpleConsumption(BaseConsumerTest.scala:85)
> {quote}
> I have recreated this with some additional trace. The tests run with a very 
> small metadata expiry interval, triggering metadata updates quite often. If a 
> metadata request immediately following a {{commitAsync()}} call creates a new 
> SSL/SASL connection, {{ConsumerNetworkClient.poll}} returns to process the 
> connection handshake packets. Since {{ConsumerNetworkClient.poll}} discards 
> all unsent packets before returning from poll, this can result in the failure 
> of the commit - the callback is invoked with {{SendFailedException}}.
> I understand that {{ConsumerNetworkClient.poll()}} discards unsent packets 
> rather than buffer them to keep the code simple. And perhaps it is ok to fail 
> {{commitAsync}} occasionally since the callback does indicate that the caller 
> should retry. But it feels like an unnecessary limitation that requires error 
> handling in client applications when there are no real failures and makes it 
> much harder to test reliably. As special handling to fix issues like 
> KAFKA-3412, KAFKA-2672 adds more complexity to the code anyway, and because 
> it is much harder to debug failures that affect only SSL/SASL, it may be 
> worth considering improving this behaviour.
> I will see if I can submit a PR for the specific issue I was seeing with the 
> impact of handshakes on {{commitAsync()}}, but I will be interested in views 
> on improving the logic in {{ConsumerNetworkClient}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-3488) commitAsync() fails if metadata update creates new SASL/SSL connection

2016-03-31 Thread Rajini Sivaram (JIRA)

Rajini Sivaram created KAFKA-3488:
-

 Summary: commitAsync() fails if metadata update creates new 
SASL/SSL connection
 Key: KAFKA-3488
 URL: https://issues.apache.org/jira/browse/KAFKA-3488
 Project: Kafka
  Issue Type: Bug
  Components: consumer
Affects Versions: 0.9.0.1
Reporter: Rajini Sivaram
Assignee: Rajini Sivaram


Sasl/SslConsumerTest.testSimpleConsumption() fails intermittently with a 
failure in {{commitAsync()}}. The exception stack trace shows:

{quote}
kafka.api.SaslPlaintextConsumerTest.testSimpleConsumption FAILED
java.lang.AssertionError: expected:<1> but was:<0>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:645)
at org.junit.Assert.assertEquals(Assert.java:631)
at 
kafka.api.BaseConsumerTest.awaitCommitCallback(BaseConsumerTest.scala:340)
at 
kafka.api.BaseConsumerTest.testSimpleConsumption(BaseConsumerTest.scala:85)
{quote}

I have recreated this with some additional trace. The tests run with a very 
small metadata expiry interval, triggering metadata updates quite often. If a 
metadata request immediately following a {{commitAsync()}} call creates a new 
SSL/SASL connection, {{ConsumerNetworkClient.poll}} returns to process the 
connection handshake packets. Since {{ConsumerNetworkClient.poll}} discards all 
unsent packets before returning from poll, this can result in the failure of 
the commit - the callback is invoked with {{SendFailedException}}.

I understand that {{ConsumerNetworkClient.poll()}} discards unsent packets 
rather than buffer them to keep the code simple. And perhaps it is ok to fail 
{{commitAsync}} occasionally since the callback does indicate that the caller 
should retry. But it feels like an unnecessary limitation that requires error 
handling in client applications when there are no real failures and makes it 
much harder to test reliably. As special handling to fix issues like 
KAFKA-3412, KAFKA-2672 adds more complexity to the code anyway, and because it 
is much harder to debug failures that affect only SSL/SASL, it may be worth 
considering improving this behaviour.

I will see if I can submit a PR for the specific issue I was seeing with the 
impact of handshakes on {{commitAsync()}}, but I will be interested in views on 
improving the logic in {{ConsumerNetworkClient}}.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-3453) Transient test failures due to MiniKDC port allocation strategy

2016-03-31 Thread Ismael Juma (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-3453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-3453.

   Resolution: Fixed
Fix Version/s: (was: 0.10.0.1)
   0.10.0.0

Fixed as part of KAFKA-3475.

> Transient test failures due to MiniKDC port allocation strategy
> ---
>
> Key: KAFKA-3453
> URL: https://issues.apache.org/jira/browse/KAFKA-3453
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.1
>Reporter: Ewen Cheslack-Postava
>Assignee: Ismael Juma
> Fix For: 0.10.0.0
>
>
> A number of tests, especially our consumer tests, fail transiently because 
> MiniKDC allocates ports by creating a socket, getting its port, then closing 
> it. As previously addressed in our own code, this causes problems because 
> that port can be reallocated before the process has a chance to bind a new 
> socket -- whether due to another test running in parallel or another process 
> simply binding the port first. This results in errors like this in the tests:
> {quote}
> java.net.BindException: Address already in use
>   at sun.nio.ch.Net.bind0(Native Method)
>   at sun.nio.ch.Net.bind(Net.java:444)
>   at sun.nio.ch.Net.bind(Net.java:436)
>   at 
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
>   at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
>   at 
> org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:198)
>   at 
> org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:51)
>   at 
> org.apache.mina.core.polling.AbstractPollingIoAcceptor.registerHandles(AbstractPollingIoAcceptor.java:547)
>   at 
> org.apache.mina.core.polling.AbstractPollingIoAcceptor.access$400(AbstractPollingIoAcceptor.java:68)
>   at 
> org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:422)
>   at 
> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> {quote}
> This is an ongoing issue that Confluent sees in its Jenkins builds, which is 
> the reason for this ticket. The real issue is actually in MiniKDC (we pass in 
> "0" for the port, but then it uses this other port allocation strategy), but 
> we either need to a) figure out a workaround or b) get a fix in upstream and 
> then update to a newer MiniKDC version.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3453) Transient test failures due to MiniKDC port allocation strategy

2016-03-31 Thread Ismael Juma (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-3453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3453:
---
Reviewer: Gwen Shapira

> Transient test failures due to MiniKDC port allocation strategy
> ---
>
> Key: KAFKA-3453
> URL: https://issues.apache.org/jira/browse/KAFKA-3453
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.1
>Reporter: Ewen Cheslack-Postava
>Assignee: Ismael Juma
> Fix For: 0.10.0.0
>
>
> A number of tests, especially our consumer tests, fail transiently because 
> MiniKDC allocates ports by creating a socket, getting its port, then closing 
> it. As previously addressed in our own code, this causes problems because 
> that port can be reallocated before the process has a chance to bind a new 
> socket -- whether due to another test running in parallel or another process 
> simply binding the port first. This results in errors like this in the tests:
> {quote}
> java.net.BindException: Address already in use
>   at sun.nio.ch.Net.bind0(Native Method)
>   at sun.nio.ch.Net.bind(Net.java:444)
>   at sun.nio.ch.Net.bind(Net.java:436)
>   at 
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
>   at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
>   at 
> org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:198)
>   at 
> org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:51)
>   at 
> org.apache.mina.core.polling.AbstractPollingIoAcceptor.registerHandles(AbstractPollingIoAcceptor.java:547)
>   at 
> org.apache.mina.core.polling.AbstractPollingIoAcceptor.access$400(AbstractPollingIoAcceptor.java:68)
>   at 
> org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:422)
>   at 
> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> {quote}
> This is an ongoing issue that Confluent sees in its Jenkins builds, which is 
> the reason for this ticket. The real issue is actually in MiniKDC (we pass in 
> "0" for the port, but then it uses this other port allocation strategy), but 
> we either need to a) figure out a workaround or b) get a fix in upstream and 
> then update to a newer MiniKDC version.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-2866) Bump up commons-collections version to 3.2.2 to address a security flaw

2016-03-31 Thread Ismael Juma (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-2866.

Resolution: Fixed
  Assignee: Ismael Juma  (was: Grant Henke)
  Reviewer: Gwen Shapira

Fixed as part of KAFKA-3475.

> Bump up commons-collections version to 3.2.2 to address a security flaw
> ---
>
> Key: KAFKA-2866
> URL: https://issues.apache.org/jira/browse/KAFKA-2866
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.0
>Reporter: Grant Henke
>Assignee: Ismael Juma
>
> Update commons-collections from 3.2.1 to 3.2.2 because of a major security 
> vulnerability. There are many other open source projects use 
> commons-collections and are also affected.
> Please see 
> http://foxglovesecurity.com/2015/11/06/what-do-weblogic-websphere-jboss-jenkins-opennms-and-your-application-have-in-common-this-vulnerability/
>  for the discovery of the vulnerability.
> https://issues.apache.org/jira/browse/COLLECTIONS-580 has the discussion 
> thread of the fix.
> https://blogs.apache.org/foundation/entry/apache_commons_statement_to_widespread
>  The ASF response to the security vulnerability.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

48 matches

Mail list logo