date:20170504


[ 
https://issues.apache.org/jira/browse/KAFKA-5161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997808#comment-15997808
 ] 

ASF GitHub Bot commented on KAFKA-5161:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2962


> reassign-partitions to check if broker of ID exists in cluster
> --
>
> Key: KAFKA-5161
> URL: https://issues.apache.org/jira/browse/KAFKA-5161
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.10.1.1
> Environment: Debian 8
>Reporter: Lawrence Weikum
>Assignee: huxi
>Priority: Minor
> Fix For: 0.11.0.0
>
>
> A topic was created with only one replica. We wanted to increase it later to 
> 3 replicas. A JSON file was created, but the IDs for the brokers were 
> incorrect and not part of the system. 
> The script or the brokers receiving the reassignment command should first 
> check if the new IDs exist in the cluster first and then continue, throwing 
> an error to the user if there is one that doesn't.
> The current effect of assign partitions to non-existant brokers is a stuck 
> replication assignment with no way to stop it. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2962: KAFKA-5161: add code in reassign-partitions to che...

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2962


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (KAFKA-5161) reassign-partitions to check if broker of ID exists in cluster


 [ 
https://issues.apache.org/jira/browse/KAFKA-5161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-5161.
--
   Resolution: Fixed
Fix Version/s: 0.11.0.0

Issue resolved by pull request 2962
[https://github.com/apache/kafka/pull/2962]

> reassign-partitions to check if broker of ID exists in cluster
> --
>
> Key: KAFKA-5161
> URL: https://issues.apache.org/jira/browse/KAFKA-5161
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.10.1.1
> Environment: Debian 8
>Reporter: Lawrence Weikum
>Assignee: huxi
>Priority: Minor
> Fix For: 0.11.0.0
>
>
> A topic was created with only one replica. We wanted to increase it later to 
> 3 replicas. A JSON file was created, but the IDs for the brokers were 
> incorrect and not part of the system. 
> The script or the brokers receiving the reassignment command should first 
> check if the new IDs exist in the cluster first and then continue, throwing 
> an error to the user if there is one that doesn't.
> The current effect of assign partitions to non-existant brokers is a stuck 
> replication assignment with no way to stop it. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: kafka-trunk-jdk8 #1484

See 


Changes:

[ismael] MINOR: Temporarily disable a few flaky tests

--
[...truncated 843.92 KB...]
kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokerListWithNoTopics STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokerListWithNoTopics PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testAutoCreateTopic STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testGetAllTopicMetadata 
STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testGetAllTopicMetadata 
PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testBasicTopicMetadata 
STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.integration.PrimitiveApiTest > testMultiProduce STARTED

kafka.integration.PrimitiveApiTest > testMultiProduce PASSED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch STARTED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch PASSED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize 
STARTED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize PASSED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests STARTED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests PASSED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch STARTED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch PASSED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression STARTED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression PASSED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic STARTED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic PASSED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest STARTED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack PASSED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopicWithCollision 
STARTED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopicWithCollision 
PASSED

kafka.integration.PlaintextTopicMetadataTest > testAliveBrokerListWithNoTopics 
STARTED

kafka.integration.PlaintextTopicMetadataTest > testAliveBrokerListWithNoTopics 
PASSED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopic STARTED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.PlaintextTopicMetadataTest > testGetAllTopicMetadata STARTED

kafka.integration.PlaintextTopicMetadataTest > testGetAllTopicMetadata PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.PlaintextTopicMetadataTest > testBasicTopicMetadata STARTED

kafka.integration.PlaintextTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.integration.MetricsDuringTopicCreationDeletionTest > 
testMetricsDuringTopicCreateDelete STARTED

kafka.integration.MetricsDuringTopicCreationDeletionTest > 
testMetricsDuringTopicCreateDelete PASSED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooLow 
STARTED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooLow PASSED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooLow 
STARTED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooLow 
PASSED

Build failed in Jenkins: kafka-trunk-jdk7 #2150

See 


Changes:

[ismael] MINOR: Temporarily disable a few flaky tests

--
[...truncated 844.95 KB...]
kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokerListWithNoTopics STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokerListWithNoTopics PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testAutoCreateTopic STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testGetAllTopicMetadata 
STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testGetAllTopicMetadata 
PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testBasicTopicMetadata 
STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.integration.PrimitiveApiTest > testMultiProduce STARTED

kafka.integration.PrimitiveApiTest > testMultiProduce PASSED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch STARTED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch PASSED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize 
STARTED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize PASSED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests STARTED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests PASSED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch STARTED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch PASSED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression STARTED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression PASSED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic STARTED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic PASSED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest STARTED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack PASSED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopicWithCollision 
STARTED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopicWithCollision 
PASSED

kafka.integration.PlaintextTopicMetadataTest > testAliveBrokerListWithNoTopics 
STARTED

kafka.integration.PlaintextTopicMetadataTest > testAliveBrokerListWithNoTopics 
PASSED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopic STARTED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.PlaintextTopicMetadataTest > testGetAllTopicMetadata STARTED

kafka.integration.PlaintextTopicMetadataTest > testGetAllTopicMetadata PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.PlaintextTopicMetadataTest > testBasicTopicMetadata STARTED

kafka.integration.PlaintextTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.integration.MetricsDuringTopicCreationDeletionTest > 
testMetricsDuringTopicCreateDelete STARTED

kafka.integration.MetricsDuringTopicCreationDeletionTest > 
testMetricsDuringTopicCreateDelete PASSED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooLow 
STARTED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooLow PASSED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooLow 
STARTED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooLow 
PASSED

[jira] [Commented] (KAFKA-5166) Add option "dry run" to Streams application reset tool


[ 
https://issues.apache.org/jira/browse/KAFKA-5166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997793#comment-15997793
 ] 

Matthias J. Sax commented on KAFKA-5166:


No problem.
(1) Internal topics: we need to reset the offsets because they do not get 
delete when we delete the topic. Thus, when the app restarts it would otherwise 
pick up those old offset (at least this was the behavior when we created the 
reset tool. I know that there is a JIRA to actually delete the offsets when a 
topic is delete -- but I am not sure if this JIRA is resolved already. Might be 
worth to double check :))
(2) Streams internally used consumer uses the {{application.id}} as 
{{group.id}} -- it's a fixed one-to-one mapping.

> Add option "dry run" to Streams application reset tool
> --
>
> Key: KAFKA-5166
> URL: https://issues.apache.org/jira/browse/KAFKA-5166
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams, tools
>Affects Versions: 0.10.2.0
>Reporter: Matthias J. Sax
>Assignee: Bharat Viswanadham
>Priority: Minor
>  Labels: needs-kip
>
> We want to add an option to Streams application reset tool, that allow for a 
> "dry run". Ie, only prints what topics would get modified/deleted without 
> actually applying any actions.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [VOTE] KIP-143: Controller Health Metrics

2017-05-04 Thread Michael André Pearce

+1 , really good to see better operational visibility being added to the broker.

Sent from my iPhone

> On 5 May 2017, at 03:34, Ismael Juma  wrote:
> 
> Hi everyone,
> 
> It seems like the discussion has converged, so I would like to initiate the
> voting process for KIP-143: Controller Health Metrics:
> 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-143
> %3A+Controller+Health+Metrics
> 
> The vote will run for a minimum of 72 hours.
> 
> Thanks,
> Ismael

Re: [DISCUSS] KIP 141 - ProducerRecordBuilder Interface

2017-05-04 Thread Michael André Pearce

My vote would be with 2, then 3 then 1.

Could I suggest maybe an option 4.

that is option 2 but with a note that there is an intent in 1/2 years time to 
deprecate the old way (under another kip). This way books materials can be 
updated over a period, code won't be making compile warnings today. And also we 
can judge if two ways is really causing an issue or not, but does hint to users 
to start actively using the new way.

Think when the new Java clients were released they released as like a beta, it 
was a release or two after that the old clients were deprecated, and still to 
be removed.

Sent from my iPhone

> On 4 May 2017, at 21:18, Matthias J. Sax  wrote:
> 
> We can go either way. I just pointed out, what I would prefer -- it's
> also quite subjective.
> 
> The least invasive change would be to add new constructors and update
> the JavaDocs to point out the semantics of `partition` parameter.
> 
> However, I still like the builder pattern: ProducerRecord has 6
> parameters with only 2 being mandatory (topic and either key or value).
> Thus, to have a complete set of overloads we would need many more
> constructors. Right now, it feels to be "incomplete" and as if the
> offered constructors got picked "randomly".
> 
> I got convinced though, that deprecation is not strictly required for
> this change. If we go with option (2), it might be good to update the
> JavaDocs of the current API to point to the new one as "recommended to use".
> 
> 
> 
> -Matthias
> 
> 
>> On 5/3/17 10:47 PM, Ewen Cheslack-Postava wrote:
>> Stephane,
>> 
>> VOTES are really on-demand based on the author, but obviously it's good to
>> come to some level of consensus in the DISCUSS thread before initiating a
>> vote. I think the request for comments/votes on your 3 options is a
>> reasonable way to gauge current opinions.
>> 
>> For myself, I think either 1 or 3 are good options, and I think at least
>> Matthias & Jay are in agreement -- basically have one preferred, but
>> possibly support 2 approaches for awhile.
>> 
>> I think 3 is the right way to go long term -- I don't expect so many more
>> built-in fields to be added, but then again I didn't expect this much churn
>> this quickly (headers were a surprise for me). We've gotten to enough
>> parameters that a builder is more effective. It sucks a bit for existing
>> users that rely on the constructors, but a full major release cycle (at the
>> minimum) is a pretty significant window, and we can always choose to extend
>> the window longer if we want to give people more time to transition. To me,
>> the biggest problem is all the tutorials and content that we *can't*
>> control -- there's a ton of code and tutorials out there that will still
>> reference the constructors, and those will last far longer than any
>> deprecation period we put in place.
>> 
>> -Ewen
>> 
>> On Wed, May 3, 2017 at 5:46 PM, Stephane Maarek <
>> steph...@simplemachines.com.au> wrote:
>> 
>>> How do votes works?
>>> 
>>> I feel there are 3 options right here, and I’d like a pre vote before a
>>> real vote?
>>> 1) Adding constructors. Could get messy over time, especially with headers
>>> coming into play, and future possible improvement to the message format
>>> 2) Adding a builder / nicer looking API (like fluent) to help build a
>>> ProducerRecord in a safe way. Issue here are two ways of building a
>>> ProducerRecord can bring confusion
>>> 3) Same as 2), but deprecating all the constructors. May be too much of an
>>> aggressive strategy
>>> 
>>> 
>>> I’m happy to go over 2), update the docs, and tell people this is the
>>> “preferred” way. Won’t outdate all the literature on Kafka, but I feel this
>>> set people up for success in the future.
>>> Thoughts  / pre vote?
>>> 
>>> On 3/5/17, 4:20 pm, "Ewen Cheslack-Postava"  wrote:
>>> 
>>>I understand the convenience of pointing at a JIRA/PR, but can we put
>>> the
>>>concrete changes proposed in the JIRA (under "Proposed Changes"). I
>>> don't
>>>think voting on the KIP would be reasonable otherwise since the changes
>>>under vote could change arbitrarily...
>>> 
>>>I'm increasingly skeptical of adding more convenience constructors --
>>> the
>>>current patch adds timestamps, we're about to add headers as well (for
>>>core, for Connect we have
>>>https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>> 145+-+Expose+Record+Headers+in+Kafka+Connect
>>>in flight). It just continues to get messier over time.
>>> 
>>>I think builders in the right context are useful, as long as they
>>> exceed a
>>>certain number of parameters (SchemaBuilder in Connect is an artifact
>>> of
>>>that position). I don't think a transition period with 2 ways to
>>> construct
>>>an object is actually a problem -- if there's always an "all N
>>> parameters"
>>>version of the constructor, all other constructors are just convenience
>>>shortcuts, but the Builder

[jira] [Commented] (KAFKA-5137) Controlled shutdown timeout message improvement


[ 
https://issues.apache.org/jira/browse/KAFKA-5137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997782#comment-15997782
 ] 

ASF GitHub Bot commented on KAFKA-5137:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2932


> Controlled shutdown timeout message improvement
> ---
>
> Key: KAFKA-5137
> URL: https://issues.apache.org/jira/browse/KAFKA-5137
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.10.2.0
>Reporter: Dustin Cote
>Assignee: Umesh Chaudhary
>Priority: Minor
>  Labels: newbie
> Fix For: 0.11.0.0
>
>
> Currently if you fail during controlled shutdown, you can get a message that 
> says the socket.timeout.ms has expired. This config actually doesn't exist on 
> the broker. Instead, we should explicitly say if we've hit the 
> controller.socket.timeout.ms or the request.timeout.ms as it's confusing to 
> take action given the current message. I believe the relevant code is here:
> https://github.com/apache/kafka/blob/0.10.2/core/src/main/scala/kafka/server/KafkaServer.scala#L428-L454
> I'm also not sure if there's another timeout that could be hit here or 
> another reason why IOException might be thrown. In the least we should call 
> out those two configs instead of the non-existent one but if we can direct to 
> the proper one that would be even better.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (KAFKA-5137) Controlled shutdown timeout message improvement


 [ 
https://issues.apache.org/jira/browse/KAFKA-5137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-5137.

   Resolution: Fixed
Fix Version/s: 0.11.0.0

Issue resolved by pull request 2932
[https://github.com/apache/kafka/pull/2932]

> Controlled shutdown timeout message improvement
> ---
>
> Key: KAFKA-5137
> URL: https://issues.apache.org/jira/browse/KAFKA-5137
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.10.2.0
>Reporter: Dustin Cote
>Assignee: Umesh Chaudhary
>Priority: Minor
>  Labels: newbie
> Fix For: 0.11.0.0
>
>
> Currently if you fail during controlled shutdown, you can get a message that 
> says the socket.timeout.ms has expired. This config actually doesn't exist on 
> the broker. Instead, we should explicitly say if we've hit the 
> controller.socket.timeout.ms or the request.timeout.ms as it's confusing to 
> take action given the current message. I believe the relevant code is here:
> https://github.com/apache/kafka/blob/0.10.2/core/src/main/scala/kafka/server/KafkaServer.scala#L428-L454
> I'm also not sure if there's another timeout that could be hit here or 
> another reason why IOException might be thrown. In the least we should call 
> out those two configs instead of the non-existent one but if we can direct to 
> the proper one that would be even better.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2932: KAFKA-5137 : Controlled shutdown timeout message i...

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2932


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [VOTE] KIP-126 - Allow KafkaProducer to split and resend oversized batches.

2017-05-04 Thread Michael Pearce

+1



From: Joel Koshy 
Sent: Friday, May 5, 2017 4:32:42 AM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-126 - Allow KafkaProducer to split and resend oversized 
batches.

+1

On Thu, May 4, 2017 at 7:00 PM Becket Qin  wrote:

> Bump.
>
> On Tue, Apr 25, 2017 at 3:17 PM, Bill Bejeck  wrote:
>
> > +1
> >
> > On Tue, Apr 25, 2017 at 4:43 PM, Dong Lin  wrote:
> >
> > > +1 (non-binding)
> > >
> > > On Tue, Apr 25, 2017 at 12:33 PM, Becket Qin 
> > wrote:
> > >
> > > > Hi,
> > > >
> > > > I would like to start the voting on KIP-126. The KIP is intended to
> > solve
> > > > the problem that RecordTooLargeExceptions are thrown from the
> producer
> > > due
> > > > to inaccurate estimation of the compression ratio. The solution is to
> > > split
> > > > and resend the over sized batches if possible. A new metric is
> > introduced
> > > > to the producer to show the batch split rate.
> > > >
> > > > The KIP wiki is following:
> > > > *https://cwiki.apache.org/confluence/pages/viewpage.
> > > action?pageId=68715855
> > > >  > > action?pageId=68715855
> > > > >*
> > > >
> > > > We have been running a producer with this patch for some time in our
> > > mirror
> > > > maker and it looks working fine.
> > > >
> > > > Thanks,
> > > >
> > > > Jiangjie (Becket) Qin
> > > >
> > >
> >
>
--
Sent from Gmail Mobile
The information contained in this email is strictly confidential and for the 
use of the addressee only, unless otherwise indicated. If you are not the 
intended recipient, please do not read, copy, use or disclose to others this 
message or any attachment. Please also notify the sender by replying to this 
email or by telephone (+44(020 7896 0011) and then delete the email and any 
copies of it. Opinions, conclusion (etc) that do not relate to the official 
business of this company shall be understood as neither given nor endorsed by 
it. IG is a trading name of IG Markets Limited (a company registered in England 
and Wales, company number 04008957) and IG Index Limited (a company registered 
in England and Wales, company number 01190902). Registered address at Cannon 
Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited 
(register number 195355) and IG Index Limited (register number 114059) are 
authorised and regulated by the Financial Conduct Authority.

[jira] [Commented] (KAFKA-5162) Add a reference to AdminClient to docs/api.html


[ 
https://issues.apache.org/jira/browse/KAFKA-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997779#comment-15997779
 ] 

ASF GitHub Bot commented on KAFKA-5162:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2958


> Add a reference to AdminClient to docs/api.html
> ---
>
> Key: KAFKA-5162
> URL: https://issues.apache.org/jira/browse/KAFKA-5162
> Project: Kafka
>  Issue Type: Sub-task
>  Components: documentation
>Affects Versions: 0.11.0.0
>Reporter: Colin P. McCabe
>Assignee: Colin P. McCabe
> Fix For: 0.11.0.0
>
>
> Add a reference to AdminClient to docs/api.html



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (KAFKA-5162) Add a reference to AdminClient to docs/api.html


 [ 
https://issues.apache.org/jira/browse/KAFKA-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-5162.

Resolution: Fixed

Issue resolved by pull request 2958
[https://github.com/apache/kafka/pull/2958]

> Add a reference to AdminClient to docs/api.html
> ---
>
> Key: KAFKA-5162
> URL: https://issues.apache.org/jira/browse/KAFKA-5162
> Project: Kafka
>  Issue Type: Sub-task
>  Components: documentation
>Affects Versions: 0.11.0.0
>Reporter: Colin P. McCabe
>Assignee: Colin P. McCabe
> Fix For: 0.11.0.0
>
>
> Add a reference to AdminClient to docs/api.html



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2958: KAFKA-5162: Add a reference to AdminClient to docs...

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2958


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4996) Fix findbugs multithreaded correctness warnings for streams


[ 
https://issues.apache.org/jira/browse/KAFKA-4996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997772#comment-15997772
 ] 

ASF GitHub Bot commented on KAFKA-4996:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2966


> Fix findbugs multithreaded correctness warnings for streams
> ---
>
> Key: KAFKA-4996
> URL: https://issues.apache.org/jira/browse/KAFKA-4996
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Colin P. McCabe
>Assignee: Amit Daga
>  Labels: newbie
>
> Fix findbugs multithreaded correctness warnings for streams
> {code}
> Multithreaded correctness Warnings
>   
>   
> 
>   
>   
>   
> 
>Code Warning   
>   
>   
> 
>AT   Sequence of calls to java.util.concurrent.ConcurrentHashMap may not 
> be atomic in 
> org.apache.kafka.streams.state.internals.Segments.getOrCreateSegment(long, 
> ProcessorContext) 
> 
>IS   Inconsistent synchronization of 
> org.apache.kafka.streams.KafkaStreams.stateListener; locked 66% of time   
>   
>   
> 
>IS   Inconsistent synchronization of 
> org.apache.kafka.streams.processor.internals.StreamThread.stateListener; 
> locked 66% of time
>   
>  
>IS   Inconsistent synchronization of 
> org.apache.kafka.streams.processor.TopologyBuilder.applicationId; locked 50% 
> of time   
>   
>  
>IS   Inconsistent synchronization of 
> org.apache.kafka.streams.state.internals.CachingKeyValueStore.context; locked 
> 66% of time   
>   
> 
>IS   Inconsistent synchronization of 
> org.apache.kafka.streams.state.internals.CachingWindowStore.cache; locked 60% 
> of time   
>   
> 
>IS   Inconsistent synchronization of 
> org.apache.kafka.streams.state.internals.CachingWindowStore.context; locked 
> 66% of time   
>   
>   
>IS   Inconsistent synchronization of 
> org.apache.kafka.streams.state.internals.CachingWindowStore.name; locked 60% 
> of time   
>   
>  
>IS   Inconsistent synchronization of 
> org.apache.kafka.streams.state.internals.CachingWindowStore.serdes; locked 
> 70% of time   
>   
>
>IS   Inconsistent synchronization of 
> org.apache.kafka.streams.state.internals.RocksDBStore.db; locked 63% of time  
>   
>   
> 
>IS   Inconsistent synchronization of 
> org.apache.kafka.streams.state.internals.RocksDBStore.serdes; locked 76% of 
> time  
>   
>   
> {code}



--
This

[GitHub] kafka pull request #2966: KAFKA-4996: Fix findbugs multithreaded correctness...

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2966


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2978: MINOR: Temporarily disable a few flaky tests

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2978


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [VOTE] KIP-118: Drop Support for Java 7 in Kafka 0.11

Hearing no objections, I have updated the release plan.

Ismael

On Tue, May 2, 2017 at 10:07 PM, Ismael Juma  wrote:

> Hi all,
>
> As I looked deeper into the details of the impact of this change, it
> became apparent to me that it may better to postpone it to a future
> release. Reasons follow:
>
> 1. Once we move to Java 8, there will be a ripple effect in all projects
> that depend on the Kafka clients jar. Given where we are in the release
> cycle[2], we would prefer to focus our resources on ensuring a high quality
> release despite the extensive changes to core parts of Kafka (message
> format, replication protocol, single-threaded controller, transactions,
> etc.). In other words, improving test coverage, stabilising flaky tests (as
> they may be hiding real bugs), fixing issues identified, etc. is more
> valuable to our users than bumping the minimum JDK version.
>
> 2. If we bump the version, exactly-once (idempotent producer and
> transactions) and headers won't be available for clients that are stuck
> using Java 7. In addition, taking advantage of KIP-101 (an important
> improvement to the replication protocol) would require down conversion (and
> consequent performance impact) if there are Java 7 clients. Users tend to
> have less control on client environments and upgrading the JDK version is
> challenging. Our research indicates that this is not uncommon.
>
> 3. Even though Oracle no longer publishes security fixes for Java 7 freely
> (a support contract is required), Red Hat continues to publish them and
> will do so until June 2018[1].
>
> Given the above, I suggest we postpone the Java 8 switch to a subsequent
> major release. It's a bit frustrating, but I think this is the right
> decision for our
> users. Are there any objections?
>
> Ismael
>
> [1] https://access.redhat.com/articles/1299013
> [2] https://cwiki.apache.org/confluence/display/KAFKA/
> Release+Plan+0.11.0.0
>
> On Tue, Feb 28, 2017 at 4:11 AM, Becket Qin  wrote:
>
>> +1
>>
>> On Mon, Feb 27, 2017 at 6:38 PM, Ismael Juma  wrote:
>>
>> > Thanks to everyone who voted and provided feedback. +1 (binding) from me
>> > too.
>> >
>> > The vote has passed with 4 binding votes (Grant, Jason, Guozhang,
>> Ismael)
>> > and 11 non-binding votes (Bill, Damian, Eno, Edoardo, Mickael, Bharat,
>> > Onur, Vahid, Colin, Apurva, Tom). There were no 0 or -1 votes.
>> >
>> > I have updated the relevant wiki pages.
>> >
>> > Ismael
>> >
>> > On Thu, Feb 9, 2017 at 3:31 PM, Ismael Juma  wrote:
>> >
>> > > Hi everyone,
>> > >
>> > > Since everyone in the discuss thread was in favour (10 people
>> responded),
>> > > I would like to initiate the voting process for KIP-118: Drop Support
>> for
>> > > Java 7 in Kafka 0.11:
>> > >
>> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> > > 118%3A+Drop+Support+for+Java+7+in+Kafka+0.11
>> > >
>> > > The vote will run for a minimum of 72 hours.
>> > >
>> > > Thanks,
>> > > Ismael
>> > >
>> > >
>> >
>>
>
>

Re: [DISCUSS] KIP-144: Exponential backoff for broker reconnect attempts

Thanks for the feedback Gwen and Colin. I agree that the original formula
was not intuitive. I updated it to include a max jitter as was suggested. I
also updated the config name to include `ms`:

https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=69408222=3=1

If there are no other concerns, I will start the vote tomorrow.

Ismael

On Mon, May 1, 2017 at 6:18 PM, Colin McCabe  wrote:

> Thanks for the KIP, Ismael & Dana!  This could be pretty important for
> avoiding congestion collapse when there are a lot of clients.
>
> It seems like a good idea to keep the "ms" suffix, like we have with
> "reconnect.backoff.ms".  So maybe we should use
> "reconnect.backoff.max.ms"?  In general unitless timeouts can be the
> source of a lot of confusion (is it seconds, milliseconds, etc.?)
>
> It's good that the KIP inject random delays (jitter) into the timeout.
> As per Gwen's point, does it make sense to put an upper bound on the
> jitter, though?  If someone sets reconnect.backoff.max to 5 minutes,
> they probably would be a little surprised to find it doing three retries
> after 100 ms in a row (as it could under the current scheme.)  Maybe a
> maximum jitter configuration would help address that, and make the
> behavior a little more intuitive.
>
> best,
> Colin
>
>
> On Thu, Apr 27, 2017, at 09:39, Gwen Shapira wrote:
> > This is a great suggestion. I like how we just do it by default instead
> > of
> > making it a choice users need to figure out.
> > Avoiding connection storms is great.
> >
> > One concern. If I understand the formula for effective maximum backoff
> > correctly, then with default maximum of 1000ms and default backoff of
> > 100ms, the effective maximum backoff will be 450ms rather than 1000ms.
> > This
> > isn't exactly intuitive.
> > I'm wondering if it makes more sense to allow "one last doubling" which
> > may
> > bring us slightly over the maximum, but much closer to it. I.e. have the
> > effective maximum be in [max.backoff - backoff, max.backoff + backoff]
> > range rather than half that. Does that make sense?
> >
> > Gwen
> >
> > On Thu, Apr 27, 2017 at 9:06 AM, Ismael Juma  wrote:
> >
> > > Hi all,
> > >
> > > Dana Powers posted a PR a while back for exponential backoff for broker
> > > reconnect attempts. Because it adds a config, a KIP is required and
> Dana
> > > seems to be busy so I posted it:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 144%3A+Exponential+backoff+for+broker+reconnect+attempts
> > >
> > > Please take a look. Your feedback is appreciated.
> > >
> > > Thanks,
> > > Ismael
> > >
> >
> >
> >
> > --
> > *Gwen Shapira*
> > Product Manager | Confluent
> > 650.450.2760 | @gwenshap
> > Follow us: Twitter  | blog
> > 
>

Re: [VOTE] KIP-126 - Allow KafkaProducer to split and resend oversized batches.

2017-05-04 Thread Joel Koshy

+1

On Thu, May 4, 2017 at 7:00 PM Becket Qin  wrote:

> Bump.
>
> On Tue, Apr 25, 2017 at 3:17 PM, Bill Bejeck  wrote:
>
> > +1
> >
> > On Tue, Apr 25, 2017 at 4:43 PM, Dong Lin  wrote:
> >
> > > +1 (non-binding)
> > >
> > > On Tue, Apr 25, 2017 at 12:33 PM, Becket Qin 
> > wrote:
> > >
> > > > Hi,
> > > >
> > > > I would like to start the voting on KIP-126. The KIP is intended to
> > solve
> > > > the problem that RecordTooLargeExceptions are thrown from the
> producer
> > > due
> > > > to inaccurate estimation of the compression ratio. The solution is to
> > > split
> > > > and resend the over sized batches if possible. A new metric is
> > introduced
> > > > to the producer to show the batch split rate.
> > > >
> > > > The KIP wiki is following:
> > > > *https://cwiki.apache.org/confluence/pages/viewpage.
> > > action?pageId=68715855
> > > >  > > action?pageId=68715855
> > > > >*
> > > >
> > > > We have been running a producer with this patch for some time in our
> > > mirror
> > > > maker and it looks working fine.
> > > >
> > > > Thanks,
> > > >
> > > > Jiangjie (Becket) Qin
> > > >
> > >
> >
>
-- 
Sent from Gmail Mobile

0.11.0.0 Release Update

Hi all,

We're quickly approaching our next time-based release. If you missed any of
the updates on the new time-based releases we'll be following, see
https://cwiki.apache.org/confluence/display/KAFKA/Time+Based+Release+Plan
for an explanation.

The release plan can be found in the usual location:
https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+0.11.0.0

Here are the important dates (also documented in the wiki):

- KIP Freeze: May 10, 2017 (a KIP must be accepted by this date in order
to be considered for this release)
- Feature Freeze: May 17, 2017 (major features merged & working on
stabilization, minor features have PR, release branch cut; anything not in
this state will be automatically moved to the next release in JIRA)
- Code Freeze: May 31, 2017 (first RC created now)
- Release: June 14, 2017

There are a couple of changes based on Ewen's feedback as release manager
for 0.10.2.0:

1. We now have a KIP freeze one week before the feature freeze to avoid
the risky and confusing situation where some KIPs are being discussed,
voted on and merged all in the same week.
2. All the dates were moved from Friday to Wednesday so that release
management doesn't spill over to the weekend.

KIPs: we have 24 adopted with 10 already committed and 10 with patches in
flight. The feature freeze is 12 days away so we have a lot of reviewing to
do, but significant changes have been merged already.

Open JIRAs: As usual, we have a lot!

*https://issues.apache.org/jira/issues/?jql=project%20%3D%20KAFKA%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened%2C%20%22Patch%20Available%22)%20AND%20fixVersion%20%3D%200.11.0.0%20ORDER%20BY%20priority%20DESC
*

146 at the moment. As we get nearer to the feature freeze, I will start
moving JIRAs out of this release.

* Closed JIRAs: So far ~191 closed tickets for 0.11.0.0:
https://issues.apache.org/jira/issues/?jql=project%20%3D%20KAFKA%20AND%20status%20in%20(Resolved%2C%20Closed)%20AND%20fixVersion%20%3D%200.11.0.0%20ORDER%20BY%20priority%20DESC

* Release features:
https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+0.11.0.0 has
a "Release Features" section that will be included with the release
notes/email for the release. I added some items to get it going. Please add
to
this list anything you think is worth noting.

I'll plan to give another update next week just before the KIP freeze.

Ismael

[jira] [Updated] (KAFKA-4763) Handle disk failure for JBOD (KIP-112)


 [ 
https://issues.apache.org/jira/browse/KAFKA-4763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-4763:
---
Status: Patch Available  (was: Open)

> Handle disk failure for JBOD (KIP-112)
> --
>
> Key: KAFKA-4763
> URL: https://issues.apache.org/jira/browse/KAFKA-4763
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Dong Lin
>Assignee: Dong Lin
>
> See 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-112%3A+Handle+disk+failure+for+JBOD
>  for motivation and design.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[VOTE] KIP-143: Controller Health Metrics

Hi everyone,

It seems like the discussion has converged, so I would like to initiate the
voting process for KIP-143: Controller Health Metrics:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-143
%3A+Controller+Health+Metrics

The vote will run for a minimum of 72 hours.

Thanks,
Ismael

[DISCUSS] KIP-133: List and Alter Configs Admin APIs

Hi all,

We've posted "KIP-133: List and Alter Configs Admin APIs" for discussion:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-133%3A+List+and+Alter+Configs+Admin+APIs

This completes the first batch of AdminClient APIs so that topic, config
and ACL management is supported.

Please take a look. Your feedback is appreciated.

Thanks,
Ismael

Re: [VOTE] KIP-126 - Allow KafkaProducer to split and resend oversized batches.

2017-05-04 Thread Becket Qin

Bump.

On Tue, Apr 25, 2017 at 3:17 PM, Bill Bejeck  wrote:

> +1
>
> On Tue, Apr 25, 2017 at 4:43 PM, Dong Lin  wrote:
>
> > +1 (non-binding)
> >
> > On Tue, Apr 25, 2017 at 12:33 PM, Becket Qin 
> wrote:
> >
> > > Hi,
> > >
> > > I would like to start the voting on KIP-126. The KIP is intended to
> solve
> > > the problem that RecordTooLargeExceptions are thrown from the producer
> > due
> > > to inaccurate estimation of the compression ratio. The solution is to
> > split
> > > and resend the over sized batches if possible. A new metric is
> introduced
> > > to the producer to show the batch split rate.
> > >
> > > The KIP wiki is following:
> > > *https://cwiki.apache.org/confluence/pages/viewpage.
> > action?pageId=68715855
> > >  > action?pageId=68715855
> > > >*
> > >
> > > We have been running a producer with this patch for some time in our
> > mirror
> > > maker and it looks working fine.
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > >
> >
>

[jira] [Commented] (KAFKA-5177) Default quick start config prompts warnings into console

2017-05-04 Thread huxi (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-5177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997692#comment-15997692
 ] 

huxi commented on KAFKA-5177:
-

Did you create topic `test` before issuing kafka--console-producer script?

> Default quick start config prompts warnings into console
> 
>
> Key: KAFKA-5177
> URL: https://issues.apache.org/jira/browse/KAFKA-5177
> Project: Kafka
>  Issue Type: Wish
>  Components: documentation
>Affects Versions: 0.10.1.0
> Environment: Mac OSX 16.5.0 Darwin Kernel Version 16.5.0 
> root:xnu-3789.51.2~3/RELEASE_X86_64 x86_64 & JDK 1.8.0_121
>Reporter: Pranas Baliuka
>
> The quick guide https://kafka.apache.org/0101/documentation.html#quickstart
> Leaves the bad first impression at the step when test messages are appended:
> {code}
> kafka_2.11-0.10.1.0 pranas$ bin/kafka-topics.sh --list --zookeeper 
> localhost:2181
> session-1
> kafka_2.11-0.10.1.0 pranas$ bin/kafka-console-producer.sh --broker-list 
> localhost:9092 --topic test
> Message 1
> [2017-05-05 09:05:10,923] WARN Error while fetching metadata with correlation 
> id 0 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
> Message 2
> Message 3
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [DISCUSS] KIP 145 - Expose Record Headers in Kafka Connect

2017-05-04 Thread Michael Pearce

Hi Ewen,

Did you get a chance to look at the updated sample showing the idea?

Did it help?

Cheers
Mike

Sent using OWA for iPhone

From: Michael Pearce 
Sent: Wednesday, May 3, 2017 10:11:55 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP 145 - Expose Record Headers in Kafka Connect

Hi Ewen,

As code I think helps, as I don’t think I explained what I meant very well.

I have pushed what I was thinking to the branch/pr.
https://github.com/apache/kafka/pull/2942

The key bits added on top here are:
new ConnectHeader that holds the header key (as string) and then header value 
object header value schema

new SubjectConverter which allows exposing a subject, in this case the subject 
is the key. - this can be used to register the header type in repos like schema 
registry, or in my case below in a property file.

We can default the subject converter to String based of Byte based where all 
header values are treated safely as String or byte[] type.

But this way you could add in your own converter which could be more 
sophisticated and convert the header based on the key.

The main part is to have access to the key, so you can look up the header value 
type, based on the key from somewhere, aka a properties file, or some central 
repo (aka schema repo), where the repo subject could be the topic + key, or 
just key if key type is global, and the schema could be primitive, String, 
byte[] or even can be more elaborate.

Cheers
Mike

On 03/05/2017, 06:00, "Ewen Cheslack-Postava"  wrote:

Michael,

Aren't JMS headers an example where the variety is a problem? Unless I'm
misunderstanding, there's not even a fixed serialization format expected
for them since JMS defines the runtime types, not the wire format. For
example, we have JMSCorrelationID (String), JMSExpires (Long), and
JMSReplyTo (Destination). These are simply run time types, so we'd need
either (a) a different serializer/deserializer for each or (b) a
serializer/deserializer that can handle all of them (e.g. Avro, JSON, etc).

What is the actual serialized format of the different fields? And if it's
not specified anywhere in the KIP, why should using the well-known type for
the header key (e.g. use StringSerializer, IntSerializer, etc) be better or
worse than using a general serialization format (e.g. Avro, JSON)? And if
the latter is the choice, how do you decide on the format?

-Ewen

On Tue, May 2, 2017 at 12:48 PM, Michael André Pearce <
michael.andre.pea...@me.com> wrote:

> Hi Ewan,
>
> So on the point of JMS the predefined/standardised JMS and JMSX headers
> have predefined types. So these can be serialised/deserialised 
accordingly.
>
> Custom jms headers agreed could be a bit more difficult but on the 80/20
> rule I would agree mostly they're string values and as anyhow you can hold
> bytes as a string it wouldn't cause any issue, defaulting to that.
>
> But I think easily we maybe able to do one better.
>
> Obviously can override the/config the headers converter but we can supply
> a default converter could take a config file with key to type mapping?
>
> Allowing people to maybe define/declare a header key with the expected
> type in some property file? To support string, byte[] and primitives? And
> undefined headers just either default to String or byte[]
>
> We could also pre define known headers like the jms ones mentioned above.
>
> E.g
>
> AwesomeHeader1=boolean
> AwesomeHeader2=long
> JMSCorrelationId=String
> JMSXGroupId=String
>
>
> What you think?
>
>
> Cheers
> Mike
>
>
>
>
>
>
> Sent from my iPhone
>
> > On 2 May 2017, at 18:45, Ewen Cheslack-Postava 
> wrote:
> >
> > A couple of thoughts:
> >
> > First, agreed that we definitely want to expose header functionality.
> Thank
> > you Mike for starting the conversation! Even if Connect doesn't do
> anything
> > special with it, there's value in being able to access/set headers.
> >
> > On motivation -- I think there are much broader use cases. When thinking
> > about exposing headers, I'd actually use Replicator as only a minor
> > supporting case. The reason is that it is a very uncommon case where
> there
> > is zero impedance mismatch between the source and sink of the data since
> > they are both Kafka. This means you don't need to think much about data
> > formats/serialization. I think the JMS use case is a better example 
since
> > JMS headers and Kafka headers don't quite match up. Here's a quick list
> of
> > use cases I can think of off the top of my head:
> >
> > 1. Include headers from other systems that support them: JMS (or really
> any
>

[jira] [Commented] (KAFKA-5166) Add option "dry run" to Streams application reset tool


[ 
https://issues.apache.org/jira/browse/KAFKA-5166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997675#comment-15997675
 ] 

Bharat Viswanadham commented on KAFKA-5166:
---

And also one more question, we are doing reset of offsets using application id, 
so here this reset tool is assuming that this application id will be used as 
group id internally by streams.
Sorry if it is naive question, could you provide any info on that..

> Add option "dry run" to Streams application reset tool
> --
>
> Key: KAFKA-5166
> URL: https://issues.apache.org/jira/browse/KAFKA-5166
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams, tools
>Affects Versions: 0.10.2.0
>Reporter: Matthias J. Sax
>Assignee: Bharat Viswanadham
>Priority: Minor
>  Labels: needs-kip
>
> We want to add an option to Streams application reset tool, that allow for a 
> "dry run". Ie, only prints what topics would get modified/deleted without 
> actually applying any actions.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5166) Add option "dry run" to Streams application reset tool


[ 
https://issues.apache.org/jira/browse/KAFKA-5166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997674#comment-15997674
 ] 

Bharat Viswanadham commented on KAFKA-5166:
---

Hi Matthias,
Thanks for clear info.. I have a question when I looked in to streams resetting 
tool, on internal topics offset reset we are doing, and next deleting the 
internal topic.
Is there any reason to do that?

> Add option "dry run" to Streams application reset tool
> --
>
> Key: KAFKA-5166
> URL: https://issues.apache.org/jira/browse/KAFKA-5166
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams, tools
>Affects Versions: 0.10.2.0
>Reporter: Matthias J. Sax
>Assignee: Bharat Viswanadham
>Priority: Minor
>  Labels: needs-kip
>
> We want to add an option to Streams application reset tool, that allow for a 
> "dry run". Ie, only prints what topics would get modified/deleted without 
> actually applying any actions.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Work started] (KAFKA-5166) Add option "dry run" to Streams application reset tool


 [ 
https://issues.apache.org/jira/browse/KAFKA-5166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-5166 started by Bharat Viswanadham.
-
> Add option "dry run" to Streams application reset tool
> --
>
> Key: KAFKA-5166
> URL: https://issues.apache.org/jira/browse/KAFKA-5166
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams, tools
>Affects Versions: 0.10.2.0
>Reporter: Matthias J. Sax
>Assignee: Bharat Viswanadham
>Priority: Minor
>  Labels: needs-kip
>
> We want to add an option to Streams application reset tool, that allow for a 
> "dry run". Ie, only prints what topics would get modified/deleted without 
> actually applying any actions.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2969: doc typo

2017-05-04 Thread smferguson

GitHub user smferguson reopened a pull request:

https://github.com/apache/kafka/pull/2969

doc typo



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/smferguson/kafka doc_typo

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2969.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2969


commit 1196ea26a960bfaaf520dde84f46c98bc3663b80
Author: Scott Ferguson 
Date:   2017-05-04T00:43:20Z

doc typo




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2969: doc typo

2017-05-04 Thread smferguson

Github user smferguson closed the pull request at:

https://github.com/apache/kafka/pull/2969


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-5154) Kafka Streams throws NPE during rebalance


[ 
https://issues.apache.org/jira/browse/KAFKA-5154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997655#comment-15997655
 ] 

Matthias J. Sax commented on KAFKA-5154:


With regard to the exception you mentioned, there is one known bug: 
https://issues.apache.org/jira/browse/KAFKA-4593 -- The exception itself 
occurs, if during restore process, the changelog end-offset changes. This 
indicates, that another task writes to the changelog topic, and thus, the task 
got migrated during restoration. Thus, we can stop restoration process and kill 
the task (as it is a zombie and does not own the partitions anymore). This 
could happen if restore takes long and a thread misses a rebalance. Just 
mentions this for clarification. As you say it's from a different application, 
it should not be related to the NPE. Btw: you can't attach any logs (there is 
just a file name but nothing more).

> Kafka Streams throws NPE during rebalance
> -
>
> Key: KAFKA-5154
> URL: https://issues.apache.org/jira/browse/KAFKA-5154
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Lukas Gemela
> Attachments: clio.txt.gz
>
>
> please see attached log, Kafka streams throws NullPointerException during 
> rebalance, which is caught by our custom exception handler
> {noformat}
> 2017-04-30T17:44:17,675 INFO  kafka-coordinator-heartbeat-thread | hades 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.coordinatorDead()
>  @618 - Marking the coordinator 10.210.200.144:9092 (id: 2147483644 rack: 
> null) dead for group hades
> 2017-04-30T17:44:27,395 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.onSuccess() 
> @573 - Discovered coordinator 10.210.200.144:9092 (id: 2147483644 rack: null) 
> for group hades.
> 2017-04-30T17:44:27,941 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinPrepare()
>  @393 - Revoking previously assigned partitions [poseidonIncidentFeed-27, 
> poseidonIncidentFeed-29, poseidonIncidentFeed-30, poseidonIncidentFeed-18] 
> for group hades
> 2017-04-30T17:44:27,947 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.sendJoinGroupRequest()
>  @407 - (Re-)joining group hades
> 2017-04-30T17:44:48,468 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.sendJoinGroupRequest()
>  @407 - (Re-)joining group hades
> 2017-04-30T17:44:53,628 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.sendJoinGroupRequest()
>  @407 - (Re-)joining group hades
> 2017-04-30T17:45:09,587 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.sendJoinGroupRequest()
>  @407 - (Re-)joining group hades
> 2017-04-30T17:45:11,961 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.onSuccess() 
> @375 - Successfully joined group hades with generation 99
> 2017-04-30T17:45:13,126 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete()
>  @252 - Setting newly assigned partitions [poseidonIncidentFeed-11, 
> poseidonIncidentFeed-27, poseidonIncidentFeed-25, poseidonIncidentFeed-29, 
> poseidonIncidentFeed-19, poseidonIncidentFeed-18] for group hades
> 2017-04-30T17:46:37,254 INFO  kafka-coordinator-heartbeat-thread | hades 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.coordinatorDead()
>  @618 - Marking the coordinator 10.210.200.144:9092 (id: 2147483644 rack: 
> null) dead for group hades
> 2017-04-30T18:04:25,993 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.onSuccess() 
> @573 - Discovered coordinator 10.210.200.144:9092 (id: 2147483644 rack: null) 
> for group hades.
> 2017-04-30T18:04:29,401 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinPrepare()
>  @393 - Revoking previously assigned partitions [poseidonIncidentFeed-11, 
> poseidonIncidentFeed-27, poseidonIncidentFeed-25, poseidonIncidentFeed-29, 
> poseidonIncidentFeed-19, poseidonIncidentFeed-18] for group hades
> 2017-04-30T18:05:10,877 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.sendJoinGroupRequest()
>  @407 - (Re-)joining group hades
> 2017-05-01T00:01:55,707 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.coordinatorDead()
>  @618 - Marking the coordinator 10.210.200.144:9092 (id: 2147483644 rack: 
> null) dead for group hades
> 2017-05-01T00:01:59,027 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.onSuccess() 
> @573 - Discovered coordinator 10.210.200.144:9092 (id: 2147483644 rack: null) 
> for

[jira] [Commented] (KAFKA-5112) Trunk compatibility tests should test against 0.10.2


[ 
https://issues.apache.org/jira/browse/KAFKA-5112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997649#comment-15997649
 ] 

ASF GitHub Bot commented on KAFKA-5112:
---

Github user cmccabe closed the pull request at:

https://github.com/apache/kafka/pull/2893


> Trunk compatibility tests should test against 0.10.2
> 
>
> Key: KAFKA-5112
> URL: https://issues.apache.org/jira/browse/KAFKA-5112
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Colin P. McCabe
>Assignee: Ismael Juma
>
> Now that 0.10.2 has been released, our trunk compatibility tests should test 
> against it.  This will ensure that 0.11 clients are backwards compatible with 
> 0.10.2 brokers.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2893: KAFKA-5112: Trunk compatibility tests should test ...

2017-05-04 Thread cmccabe

Github user cmccabe closed the pull request at:

https://github.com/apache/kafka/pull/2893


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2978: MINOR: Temporarily disable a few flaky tests

2017-05-04 Thread ijuma

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/2978

MINOR: Temporarily disable a few flaky tests



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka disable-some-flaky-tests

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2978.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2978


commit be75a37fd6db8518c9030a219409f40bfe807dc2
Author: Ismael Juma 
Date:   2017-05-04T23:33:05Z

MINOR: Temporarily disable a few flaky tests




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] KIP-140: Add administrative RPCs for adding, deleting, and listing ACLs

2017-05-04 Thread Colin McCabe

That's a good point.  It's worth mentioning that there is a KIP in
progress, KIP-133: List and Alter Configs Admin APIs, that should help
with those.

In the long term, it would be nice if we could deprecate
'allow.everyone.if.no.acl.found', along with topic auto-creation.  It
should be possible to get the functionality of
'allow.everyone.if.no.acl.found' by just adding a few ALLOW * ACLs. 
Maybe we could even do it in an upgrade script?  It will take a while to
get there.

best,
Colin


On Thu, May 4, 2017, at 15:09, dan wrote:
> what about the configs for: `allow.everyone.if.no.acl.found` and `
> super.users`?
> 
> i understand they are an implementation detail of
> `SimpleAclAuthorizer` configs,
> but without these its difficult to make sense of what permissions a
> `ListAclsResponse`
> actually represents.
> 
> maybe something for another kip.
> 
> dan
> 
> On Thu, May 4, 2017 at 2:36 PM, Colin McCabe  wrote:
> 
> > On Thu, May 4, 2017, at 13:46, Magnus Edenhill wrote:
> > > Hey Colin,
> > >
> > > good KIP!
> > >
> > > Some comments:
> > >
> > > 1a. For operation, permission_type and resource_type: is there any reason
> > > for having the any and unknown enums as negative values?
> > > Since neither of these fields has an integer significance (unlike for
> > > example offsets which use negative offsets for logical offsets) I dont
> > > really see a reason to do this. It might also trick client developers to
> > > make assumptions on future negative values (if resource_type < 0:
> > > treat_as_invalid()...), unless that's the reason :). This might be my
> > > personal preference but encoding extended meaning into types should be
> > > avoided unless needed, and I dont think it is needed for enums.
> >
> > Hi Magnus,
> >
> > That's a fair question.  I don't have a strong preference either way.
> > If it is more convenient or consistent to start at 0, we can certainly
> > do that.
> >
> > >
> > > but..
> > >
> > > 1b. Since most clients implementing the ACL requests probably wont make
> > > much use of this API themselves but rather treat it as a straight
> > > pass-through between the protocol and the client's public API to the
> > > application, could we save ourselves some work (current and future) to
> > > make
> > > the enums as nullable_strings instead of integer enums? This would cut
> > > down
> > > on the number of enum-to-str and vice versa conversions needed, and would
> > > also make the APIs more future proof since an added resource_type (et.al
> > )
> > > would not need a client, or even tool, update, and the new type will not
> > > show up as UNKNOWN but of its real value.
> > > From a broker-side verification perspective there should really be no
> > > difference since the enum values will need to be interpreted anyway.
> > > So instead of int enum { unknown = -2, any = -1, deny, allow }, we have {
> > > null, "deny", "allow" }.
> >
> > Strings use much, much more space, though.  An INT8 is one byte, whereas
> > the string "clusterAction" is 13 bytes, plus a 2 byte length field (if I
> > remember our serialization correctly)  A 10x or 15x RPC space penalty
> > starts to really hurt, especially when you have hundreds or thousands of
> > ACLs, and each one has 6 fields.
> >
> > >
> > >
> > > 2. "Each of the arguments to ListAclsRequest acts as a filter."
> > > Specify if these filters are OR:ed or AND:ed.
> >
> > Yeah, they are ANDed.  I'll add a note.
> >
> > >
> > > 3. "CreateAclsRequest and CreateAclsResponse"
> > > What happens if a client attempts to create an ACL entry which is
> > > identical
> > > to one already created in the cluster?
> > > Is this an error? silently ignored? resulting in duplicates?
> >
> > Unfortunately, we are somewhat at the mercy of the
> > kafka.security.auth.Authorizer implementation here.  The default
> > SimpleAclAuthorizer does not allow duplicates to be added.
> > If the Authorizer doesn't de-duplicate, we can't add de-duplication
> > without doing distributed locking of some sort.  I think it makes sense
> > to add a new exception, DuplicateAcl, though, and specify that the
> > authorizer ought to throw that exception.  Let me add a little more
> > detail here.
> >
> > >
> > > 4. "Compatibility plan"
> > > "If we later add new resource types, operation types, and so forth, we
> > > would like to be able to interact with them with the old AdminClient.
> > > This
> > > is why the AclResourceType, AclOperation, and AclPermissionType enums
> > > have
> > > UNKNOWN values.  If we get an INT8 which we don't know the name for, it
> > > gets mapped to an UNKNOWN object."
> > >
> > > I'm not sure who "we" are. Is it the broker or the client?
> > > (also see 1b for avoiding UNKNOWN altogether)
> >
> > Both the broker and the client can get "unknown" values for those enums,
> > if someone sends them something they don't understand.  The broker will
> > not add new ACLs that contain unknowns, nor will it delete using filters
> > that have

[jira] [Created] (KAFKA-5177) Default quick start config prompts warnings into console

2017-05-04 Thread Pranas Baliuka (JIRA)

Pranas Baliuka created KAFKA-5177:
-

 Summary: Default quick start config prompts warnings into console
 Key: KAFKA-5177
 URL: https://issues.apache.org/jira/browse/KAFKA-5177
 Project: Kafka
  Issue Type: Wish
  Components: documentation
Affects Versions: 0.10.1.0
 Environment: Mac OSX 16.5.0 Darwin Kernel Version 16.5.0 
root:xnu-3789.51.2~3/RELEASE_X86_64 x86_64 & JDK 1.8.0_121
Reporter: Pranas Baliuka


The quick guide https://kafka.apache.org/0101/documentation.html#quickstart
Leaves the bad first impression at the step when test messages are appended:
{code}
kafka_2.11-0.10.1.0 pranas$ bin/kafka-topics.sh --list --zookeeper 
localhost:2181
session-1
kafka_2.11-0.10.1.0 pranas$ bin/kafka-console-producer.sh --broker-list 
localhost:9092 --topic test
Message 1
[2017-05-05 09:05:10,923] WARN Error while fetching metadata with correlation 
id 0 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
Message 2
Message 3
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5166) Add option "dry run" to Streams application reset tool


[ 
https://issues.apache.org/jira/browse/KAFKA-5166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997568#comment-15997568
 ] 

Matthias J. Sax commented on KAFKA-5166:


All tools and their parameters _are_ public API -- your approach to add a new 
parameter is correct -- and thus, it changes public API as it adds something to 
it :)

> Add option "dry run" to Streams application reset tool
> --
>
> Key: KAFKA-5166
> URL: https://issues.apache.org/jira/browse/KAFKA-5166
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams, tools
>Affects Versions: 0.10.2.0
>Reporter: Matthias J. Sax
>Assignee: Bharat Viswanadham
>Priority: Minor
>  Labels: needs-kip
>
> We want to add an option to Streams application reset tool, that allow for a 
> "dry run". Ie, only prints what topics would get modified/deleted without 
> actually applying any actions.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5173) SASL tests failing with Could not find a 'KafkaServer' or 'sasl_plaintext.KafkaServer' entry in the JAAS configuration


[ 
https://issues.apache.org/jira/browse/KAFKA-5173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997561#comment-15997561
 ] 

Ismael Juma commented on KAFKA-5173:


Another somewhat recent change was 
https://github.com/apache/kafka/commit/63f01a8af897ee9510ab0364f2d026e4bf5472ef 
 

> SASL tests failing with Could not find a 'KafkaServer' or 
> 'sasl_plaintext.KafkaServer' entry in the JAAS configuration
> --
>
> Key: KAFKA-5173
> URL: https://issues.apache.org/jira/browse/KAFKA-5173
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ismael Juma
> Fix For: 0.11.0.0
>
>
> I've seen this a few times. One example:
> {code}
> java.lang.IllegalArgumentException: Could not find a 'KafkaServer' or 
> 'sasl_plaintext.KafkaServer' entry in the JAAS configuration. System property 
> 'java.security.auth.login.config' is /tmp/kafka8162725028002772063.tmp
>   at 
> org.apache.kafka.common.security.JaasContext.defaultContext(JaasContext.java:131)
>   at 
> org.apache.kafka.common.security.JaasContext.load(JaasContext.java:96)
>   at 
> org.apache.kafka.common.security.JaasContext.load(JaasContext.java:78)
>   at 
> org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:100)
>   at 
> org.apache.kafka.common.network.ChannelBuilders.serverChannelBuilder(ChannelBuilders.java:73)
>   at kafka.network.Processor.(SocketServer.scala:423)
>   at kafka.network.SocketServer.newProcessor(SocketServer.scala:145)
>   at 
> kafka.network.SocketServer$$anonfun$startup$1$$anonfun$apply$1.apply$mcVI$sp(SocketServer.scala:96)
>   at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160)
>   at 
> kafka.network.SocketServer$$anonfun$startup$1.apply(SocketServer.scala:95)
>   at 
> kafka.network.SocketServer$$anonfun$startup$1.apply(SocketServer.scala:90)
>   at 
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>   at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
>   at kafka.network.SocketServer.startup(SocketServer.scala:90)
>   at kafka.server.KafkaServer.startup(KafkaServer.scala:218)
>   at kafka.utils.TestUtils$.createServer(TestUtils.scala:126)
>   at 
> kafka.integration.BaseTopicMetadataTest.setUp(BaseTopicMetadataTest.scala:51)
>   at 
> kafka.integration.SaslPlaintextTopicMetadataTest.kafka$api$SaslTestHarness$$super$setUp(SaslPlaintextTopicMetadataTest.scala:23)
>   at kafka.api.SaslTestHarness$class.setUp(SaslTestHarness.scala:31)
>   at 
> kafka.integration.SaslPlaintextTopicMetadataTest.setUp(SaslPlaintextTopicMetadataTest.scala:23)
> {code}
> https://builds.apache.org/job/kafka-trunk-jdk8/1479/testReport/junit/kafka.integration/SaslPlaintextTopicMetadataTest/testIsrAfterBrokerShutDownAndJoinsBack/
> [~rsivaram], any ideas?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Work started] (KAFKA-5170) KafkaAdminClientIntegration test should wait until metadata is propagated to all brokers


 [ 
https://issues.apache.org/jira/browse/KAFKA-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-5170 started by Colin P. McCabe.
--
> KafkaAdminClientIntegration test should wait until metadata is propagated to 
> all brokers
> 
>
> Key: KAFKA-5170
> URL: https://issues.apache.org/jira/browse/KAFKA-5170
> Project: Kafka
>  Issue Type: Bug
>  Components: admin
>Affects Versions: 0.11.0.0
>Reporter: Colin P. McCabe
>Assignee: Colin P. McCabe
> Fix For: 0.11.0.0
>
>
> The KafkaAdminClientIntegration test and its subclasses should wait until the 
> metadata is propagated to all brokers.  We have seen a few test failures that 
> resulted from some brokers having partial metadata.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Work started] (KAFKA-5159) Update Kafka documentation to mention AdminClient


 [ 
https://issues.apache.org/jira/browse/KAFKA-5159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-5159 started by Colin P. McCabe.
--
> Update Kafka documentation to mention AdminClient
> -
>
> Key: KAFKA-5159
> URL: https://issues.apache.org/jira/browse/KAFKA-5159
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Ismael Juma
>Assignee: Colin P. McCabe
>
> We should have a brief section for the AdminClient in the Kafka docs. It's OK 
> to point to the Javadoc for more information (we do the same for the new Java 
> consumer and producer).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5173) SASL tests failing with Could not find a 'KafkaServer' or 'sasl_plaintext.KafkaServer' entry in the JAAS configuration


[ 
https://issues.apache.org/jira/browse/KAFKA-5173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997529#comment-15997529
 ] 

Ismael Juma commented on KAFKA-5173:


Another one that is slightly different, but looks to be the same underlying 
cause:

{code}
[2017-05-04 20:34:00,688] FATAL Fatal error during KafkaServer startup. Prepare 
to shutdown (kafka.server.KafkaServer:118)
java.lang.SecurityException: zookeeper.set.acl is true, but the verification of 
the JAAS login file failed.
at kafka.server.KafkaServer.initZk(KafkaServer.scala:322)
at kafka.server.KafkaServer.startup(KafkaServer.scala:190)
at kafka.utils.TestUtils$.createServer(TestUtils.scala:126)
at 
kafka.integration.KafkaServerTestHarness.$anonfun$setUp$1(KafkaServerTestHarness.scala:91)
at 
scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:234)
at scala.collection.Iterator.foreach(Iterator.scala:929)
at scala.collection.Iterator.foreach$(Iterator.scala:929)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1417)
at scala.collection.IterableLike.foreach(IterableLike.scala:71)
at scala.collection.IterableLike.foreach$(IterableLike.scala:70)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at scala.collection.TraversableLike.map(TraversableLike.scala:234)
at scala.collection.TraversableLike.map$(TraversableLike.scala:227)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at 
kafka.integration.KafkaServerTestHarness.setUp(KafkaServerTestHarness.scala:91)
at 
kafka.api.IntegrationTestHarness.setUp(IntegrationTestHarness.scala:65)
at 
kafka.api.EndToEndAuthorizationTest.setUp(EndToEndAuthorizationTest.scala:158)
at 
kafka.api.SaslEndToEndAuthorizationTest.setUp(SaslEndToEndAuthorizationTest.scala:41)
{code}

https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/3497/testReport/kafka.api/SaslGssapiSslEndToEndAuthorizationTest/testNoProduceWithoutDescribeAcl/

[~baluchicken], this typically happens if the cached `Configuration` instance 
returned by `Configuration.getConfiguration()` is the wrong one.

> SASL tests failing with Could not find a 'KafkaServer' or 
> 'sasl_plaintext.KafkaServer' entry in the JAAS configuration
> --
>
> Key: KAFKA-5173
> URL: https://issues.apache.org/jira/browse/KAFKA-5173
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ismael Juma
> Fix For: 0.11.0.0
>
>
> I've seen this a few times. One example:
> {code}
> java.lang.IllegalArgumentException: Could not find a 'KafkaServer' or 
> 'sasl_plaintext.KafkaServer' entry in the JAAS configuration. System property 
> 'java.security.auth.login.config' is /tmp/kafka8162725028002772063.tmp
>   at 
> org.apache.kafka.common.security.JaasContext.defaultContext(JaasContext.java:131)
>   at 
> org.apache.kafka.common.security.JaasContext.load(JaasContext.java:96)
>   at 
> org.apache.kafka.common.security.JaasContext.load(JaasContext.java:78)
>   at 
> org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:100)
>   at 
> org.apache.kafka.common.network.ChannelBuilders.serverChannelBuilder(ChannelBuilders.java:73)
>   at kafka.network.Processor.(SocketServer.scala:423)
>   at kafka.network.SocketServer.newProcessor(SocketServer.scala:145)
>   at 
> kafka.network.SocketServer$$anonfun$startup$1$$anonfun$apply$1.apply$mcVI$sp(SocketServer.scala:96)
>   at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160)
>   at 
> kafka.network.SocketServer$$anonfun$startup$1.apply(SocketServer.scala:95)
>   at 
> kafka.network.SocketServer$$anonfun$startup$1.apply(SocketServer.scala:90)
>   at 
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>   at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
>   at kafka.network.SocketServer.startup(SocketServer.scala:90)
>   at kafka.server.KafkaServer.startup(KafkaServer.scala:218)
>   at kafka.utils.TestUtils$.createServer(TestUtils.scala:126)
>   at 
> kafka.integration.BaseTopicMetadataTest.setUp(BaseTopicMetadataTest.scala:51)
>   at 
> kafka.integration.SaslPlaintextTopicMetadataTest.kafka$api$SaslTestHarness$$super$setUp(SaslPlaintextTopicMetadataTest.scala:23)
>   at kafka.api.SaslTestHarness$class.setUp(SaslTestHarness.scala:31)
>   at 
> kafka.integration.SaslPlaintextTopicMetadataTest.setUp(SaslPlaintextTopicMetadataTest.scala:23)
> {code}
> https://builds.apache.org/job/kafka-trunk-jdk8/1479/testReport/junit/kafka.integration/SaslPlaintextTopicMetadataTest/testIsrAfterBrokerShutDownAndJoinsBack/
> [~rsivaram], any ideas?



--

[jira] [Assigned] (KAFKA-5176) AdminClient: add controller and clusterId methods to DescribeClusterResults


 [ 
https://issues.apache.org/jira/browse/KAFKA-5176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin P. McCabe reassigned KAFKA-5176:
--

Assignee: Colin P. McCabe

> AdminClient: add controller and clusterId methods to DescribeClusterResults
> ---
>
> Key: KAFKA-5176
> URL: https://issues.apache.org/jira/browse/KAFKA-5176
> Project: Kafka
>  Issue Type: Improvement
>  Components: admin
>Affects Versions: 0.11.0.0
>Reporter: Colin P. McCabe
>Assignee: Colin P. McCabe
> Fix For: 0.11.0.0
>
>
> AdminClient: add controller and clusterId methods to DescribeClusterResults



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5176) AdminClient: add controller and clusterId methods to DescribeClusterResults


[ 
https://issues.apache.org/jira/browse/KAFKA-5176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997528#comment-15997528
 ] 

ASF GitHub Bot commented on KAFKA-5176:
---

GitHub user cmccabe opened a pull request:

https://github.com/apache/kafka/pull/2977

KAFKA-5176: AdminClient: add controller and clusterId methods to Desc…

…ribeClusterResults

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cmccabe/kafka KAFKA-5176

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2977.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2977


commit 07ceddfe1920e2d733a653fe6d9ae61dfd6443c2
Author: Colin P. Mccabe 
Date:   2017-05-04T22:20:49Z

KAFKA-5176: AdminClient: add controller and clusterId methods to 
DescribeClusterResults




> AdminClient: add controller and clusterId methods to DescribeClusterResults
> ---
>
> Key: KAFKA-5176
> URL: https://issues.apache.org/jira/browse/KAFKA-5176
> Project: Kafka
>  Issue Type: Improvement
>  Components: admin
>Affects Versions: 0.11.0.0
>Reporter: Colin P. McCabe
> Fix For: 0.11.0.0
>
>
> AdminClient: add controller and clusterId methods to DescribeClusterResults



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2977: KAFKA-5176: AdminClient: add controller and cluste...

2017-05-04 Thread cmccabe

GitHub user cmccabe opened a pull request:

https://github.com/apache/kafka/pull/2977

KAFKA-5176: AdminClient: add controller and clusterId methods to Descâ¦

â¦ribeClusterResults

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cmccabe/kafka KAFKA-5176

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2977.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2977


commit 07ceddfe1920e2d733a653fe6d9ae61dfd6443c2
Author: Colin P. Mccabe 
Date:   2017-05-04T22:20:49Z

KAFKA-5176: AdminClient: add controller and clusterId methods to 
DescribeClusterResults




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

2017-05-04 Thread Kyle Winkelman

I sure can. I have added the following description to my KIP. If this
doesn't help let me know and I will take some more time to build a diagram
and make more of a step by step description:

Example with Current API:

KTable table1 =
builder.stream("topic1").groupByKey().aggregate(initializer1, aggregator1,
aggValueSerde1, storeName1);
KTable table2 =
builder.stream("topic2").groupByKey().aggregate(initializer2, aggregator2,
aggValueSerde2, storeName2);
KTable table3 =
builder.stream("topic3").groupByKey().aggregate(initializer3, aggregator3,
aggValueSerde3, storeName3);
KTable cogrouped = table1.outerJoin(table2,
joinerOneAndTwo).outerJoin(table3, joinerOneTwoAndThree);

As you can see this creates 3 StateStores, requires 3 initializers, and 3
aggValueSerdes. This also adds the pressure to user to define what the
intermediate values are going to be (V1, V2, V3). They are left with a
couple choices, first to make V1, V2, and V3 all the same as CG and the two
joiners are more like mergers, or second make them intermediate states such
as Topic1Map, Topic2Map, and Topic3Map and the joiners use those to build
the final aggregate CG value. This is something the user could avoid
thinking about with this KIP.

When a new input arrives lets say at "topic1" it will first go through a
KStreamAggregate grabbing the current aggregate from storeName1. It will
produce this in the form of the first intermediate value and get sent
through a KTableKTableOuterJoin where it will look up the current value of
the key in storeName2. It will use the first joiner to calculate the second
intermediate value, which will go through an additional
KTableKTableOuterJoin. Here it will look up the current value of the key in
storeName3 and use the second joiner to build the final aggregate value.

If you think through all possibilities for incoming topics you will see
that no matter which topic it comes in through all three stores are queried
and all of the joiners must get used.

Topology wise for N incoming streams this creates N
KStreamAggregates, 2*(N-1) KTableKTableOuterJoins, and N-1
KTableKTableJoinMergers.

Example with Proposed API:

KGroupedStream grouped1 = builder.stream("topic1").groupByKey();
KGroupedStream grouped2 = builder.stream("topic2").groupByKey();
KGroupedStream grouped3 = builder.stream("topic3").groupByKey();
KTable cogrouped = grouped1.cogroup(initializer1, aggregator1,
aggValueSerde1, storeName1)
.cogroup(grouped2, aggregator2)
.cogroup(grouped3, aggregator3)
.aggregate();

As you can see this creates 1 StateStore, requires 1 initializer, and 1
aggValueSerde. The user no longer has to worry about the intermediate
values and the joiners. All they have to think about is how each stream
impacts the creation of the final CG object.

When a new input arrives lets say at "topic1" it will first go through a
KStreamAggreagte and grab the current aggregate from storeName1. It will
add its incoming object to the aggregate, update the store and pass the new
aggregate on. This new aggregate goes through the KStreamCogroup which is
pretty much just a pass through processor and you are done.

Topology wise for N incoming streams the new api will only every create N
KStreamAggregates and 1 KStreamCogroup.

On Thu, May 4, 2017 at 4:42 PM, Matthias J. Sax 
wrote:

> Kyle,
>
> thanks a lot for the KIP. Maybe I am a little slow, but I could not
> follow completely. Could you maybe add a more concrete example, like 3
> streams with 3 records each (plus expected result), and show the
> difference between current way to to implement it and the proposed API?
> This could also cover the internal processing to see what store calls
> would be required for both approaches etc.
>
> I think, it's pretty advanced stuff you propose, and it would help to
> understand it better.
>
> Thanks a lot!
>
>
> -Matthias
>
>
>
> On 5/4/17 11:39 AM, Kyle Winkelman wrote:
> > I have made a pull request. It can be found here.
> >
> > https://github.com/apache/kafka/pull/2975
> >
> > I plan to write some more unit tests for my classes and get around to
> > writing documentation for the public api additions.
> >
> > One thing I was curious about is during the
> KCogroupedStreamImpl#aggregate
> > method I pass null to the KGroupedStream#repartitionIfRequired method. I
> > can't supply the store name because if more than one grouped stream
> > repartitions an error is thrown. Is there some name that someone can
> > recommend or should I leave the null and allow it to fall back to the
> > KGroupedStream.name?
> >
> > Should this be expanded to handle grouped tables? This would be pretty
> easy
> > for a normal aggregate but one allowing session stores and windowed
> stores
> > would required KTableSessionWindowAggregate and KTableWindowAggregate
> > implementations.
> >
> > Thanks,
> > Kyle
> >
> > On May 4, 2017 1:24 PM, "Eno Thereska"

[jira] [Created] (KAFKA-5176) AdminClient: add controller and clusterId methods to DescribeClusterResults

Colin P. McCabe created KAFKA-5176:
--

 Summary: AdminClient: add controller and clusterId methods to 
DescribeClusterResults
 Key: KAFKA-5176
 URL: https://issues.apache.org/jira/browse/KAFKA-5176
 Project: Kafka
  Issue Type: Improvement
  Components: admin
Affects Versions: 0.11.0.0
Reporter: Colin P. McCabe
 Fix For: 0.11.0.0


AdminClient: add controller and clusterId methods to DescribeClusterResults



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: kafka-trunk-jdk8 #1483

See 


Changes:

[wangguoz] KAFKA-5104: DumpLogSegments should not open index files with `rw`

--
[...truncated 1.49 MB...]

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliest PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPersistEpochsBetweenInstances STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPersistEpochsBetweenInstances PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotClearAnythingIfOffsetToFirstOffset STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotClearAnythingIfOffsetToFirstOffset PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotLetOffsetsGoBackwardsEvenIfEpochsProgress STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotLetOffsetsGoBackwardsEvenIfEpochsProgress PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldGetFirstOffsetOfSubsequentEpochWhenOffsetRequestedForPreviousEpoch STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldGetFirstOffsetOfSubsequentEpochWhenOffsetRequestedForPreviousEpoch PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest2 STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest2 PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearEarliestOnEmptyCache 
STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearEarliestOnEmptyCache 
PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPreserveResetOffsetOnClearEarliestIfOneExists STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPreserveResetOffsetOnClearEarliestIfOneExists PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnInvalidOffsetIfEpochIsRequestedWhichIsNotCurrentlyTracked STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnInvalidOffsetIfEpochIsRequestedWhichIsNotCurrentlyTracked PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldFetchEndOffsetOfEmptyCache 
STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldFetchEndOffsetOfEmptyCache 
PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliestAndUpdateItsOffset STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliestAndUpdateItsOffset PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearAllEntries STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearAllEntries PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearLatestOnEmptyCache 
STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearLatestOnEmptyCache 
PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryHeadIfUndefinedPassed STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryHeadIfUndefinedPassed PASSED

kafka.server.ReplicationQuotaManagerTest > shouldThrottleOnlyDefinedReplicas 
STARTED

kafka.server.ReplicationQuotaManagerTest > shouldThrottleOnlyDefinedReplicas 
PASSED

kafka.server.ReplicationQuotaManagerTest > 
shouldSupportWildcardThrottledReplicas STARTED

kafka.server.ReplicationQuotaManagerTest > 
shouldSupportWildcardThrottledReplicas PASSED

kafka.server.ReplicationQuotaManagerTest > 
shouldExceedQuotaThenReturnBackBelowBoundAsTimePasses STARTED

kafka.server.ReplicationQuotaManagerTest > 
shouldExceedQuotaThenReturnBackBelowBoundAsTimePasses PASSED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestWithUnsupportedVersion STARTED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestWithUnsupportedVersion PASSED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestBeforeSaslHandshakeRequest STARTED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestBeforeSaslHandshakeRequest PASSED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestAfterSaslHandshakeRequest STARTED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestAfterSaslHandshakeRequest PASSED

kafka.server.ServerMetricsTest > testMetricsConfig STARTED

kafka.server.ServerMetricsTest > testMetricsConfig PASSED

kafka.server.ProduceRequestTest > testSimpleProduceRequest STARTED

kafka.server.ProduceRequestTest > testSimpleProduceRequest PASSED

kafka.server.ProduceRequestTest > testCorruptLz4ProduceRequest STARTED

kafka.server.ProduceRequestTest > testCorruptLz4ProduceRequest PASSED

kafka.server.KafkaMetricReporterClusterIdTest > testClusterIdPresent STARTED

kafka.server.KafkaMetricReporterClusterIdTest > testClusterIdPresent PASSED

kafka.server.ApiVersionsTest > testApiVersions STARTED

Re: [DISCUSS] KIP-146: Classloading Isolation in Connect

2017-05-04 Thread Gwen Shapira

Hi,

What about Transformations? Are they going to be isolated too? If they are,
what do we do with the built-in ones?

On Sat, Apr 29, 2017 at 9:16 AM, Konstantine Karantasis <
konstant...@confluent.io> wrote:

> * Because of KIP number collision, please disregard my previous KIP
> announcement and use this thread for discussion instead *
>
>
> Hi everyone,
>
> we aim to address dependency conflicts in Kafka Connect soon by applying
> class loading isolation.
>
> Feel free to take a look at KIP-146 here:
>
> *https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 146+-+Classloading+Isolation+in+Connect
>  146+-+Classloading+Isolation+in+Connect>*
>
> which describes minimal required changes to public interfaces and the
> general implementation approach.
>
> This is a much wanted feature for Kafka Connect. Your feedback is highly
> appreciated.
>
> -Konstantine
>



-- 
*Gwen Shapira*
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter  | blog

Re: [DISCUSS] KIP-146: Classloading Isolation in Connect

2017-05-04 Thread Gwen Shapira

On Wed, May 3, 2017 at 11:17 AM, Konstantine Karantasis <
konstant...@confluent.io> wrote:

> Thanks Ewen. I'm replying inline as well.
>
> On Tue, May 2, 2017 at 11:24 AM, Ewen Cheslack-Postava 
> wrote:
>
> > Thanks for the KIP.
> >
> > A few responses inline, followed by additional comments.
> >
> > On Mon, May 1, 2017 at 9:50 PM, Konstantine Karantasis <
> > konstant...@confluent.io> wrote:
> >
> > > Gwen, Randall thank you for your very insightful observations. I'm glad
> > you
> > > find this first draft to be an adequate platform for discussion.
> > >
> > > I'll attempt replying to your comments in order.
> > >
> > > Gwen, I also debated exactly the same two options: a) interpreting
> > absence
> > > of module path as a user's intention to turn off isolation and b)
> > > explicitly using an additional boolean property. A few reasons why I
> went
> > > with b) in this first draft are:
> > > 1) As Randall mentions, to leave the option of using a default value
> > open.
> > > If not immediately in the first version of isolation, maybe in the
> > future.
> > > 2) I didn't like the implicit character of the choice of interpreting
> an
> > > empty string as a clear intention to turn isolation off by the user.
> Half
> > > the time could be just that users forget to set a location, although
> > they'd
> > > like to use class loading isolation.
> > > 3) There's a slim possibility that in rare occasions a user might want
> to
> > > avoid even the slightest increase in memory consumption due to class
> > > loading duplication. I admit this should be very rare, but given the
> > other
> > > concerns and that we would really like to keep the isolation
> > implementation
> > > simple, the option to turn off this feature by using only one
> additional
> > > config property might not seem too excessive. At least at the start of
> > this
> > > discussion.
> > > 4) Debugging during development might be simpler in some cases.
> > > 5) Finally, as you mention, this could allow for smoother upgrades.
> > >
> >
> > I'm not sure any of these keep you from removing the extra config. Is
> there
> > any reason you couldn't have clean support for relying on the CLASSPATH
> > while still supporting the classloaders? Then getting people onto the new
> > classloaders does require documentation for how to install connectors,
> but
> > that's pretty minimal. And we don't break existing installations where
> > people are just adding to the CLASSPATH. It seems like this:
> >
> > 1. Allows you to set a default. Isolation is always enabled, but we won't
> > include any paths/directories we already use. Setting a default just
> > requires specifying a new location where we'd hold these directories.
> > 2. It doesn't require the implicit choice -- you actually never turn off
> > isolation, but still support the regular CLASSPATH with an empty list of
> > isolated loaders
> > 3. The user can still use CLASSPATH if they want to minimize classloader
> > overhead
> > 4. Debugging can still use CLASSPATH
> > 5. Upgrades just work.
> >
>
> Falling back to CLASSPATH for non-isolated mode makes sense. The extra
> config property was suggested proactively, as well as for clarity and
> handling of defaults. But it's much better if we can do without it. Will be
> removed.
>
>
Thank you (and I'm sure Confluent's support team also thanks you - one
configuration should be harder to accidentally screw up than two...)



>
> >
> >
> > > Randall, regarding your comments:
> > > 1) To keep its focus narrow, this KIP, as well as the first
> > implementation
> > > of isolation in Connect, assume filesystem based discovery. With
> careful
> > > implementation, transitioning to discovery schemes that support broader
> > > URIs I believe should be easy in the future.
> > >
> >
> > Maybe just mention a couple of quick examples in the KIP. When described
> > inline it might be more obvious that it will extend cleanly.
> >
> >
> There's an example for a filesystem-based structure. I will enhance it.
>
>
>
> > > 2) The example you give makes a good point. However I'm inclined to say
> > > that such cases should be addressed more as exceptions rather than as
> > being
> > > the common case. Therefore, I wouldn't see all dependencies imported by
> > the
> > > framework as required to be filtered out, because in that case we lose
> > the
> > > advantage of isolation between the framework and the connectors (and we
> > are
> > > left only with isolation between connectors).
> >
> > 3) I tried to abstract implementation details in this the KIP, but you
> are
> > > right. Even though filtering here is mainly used semantically rather
> than
> > > literally, it gives an implementation hint that we could avoid.
> > >
> >
> > I think we're missing another option -- don't do filtering and require
> that
> > those dependencies are correctly filtered out of the modules. If we want
> to
> > be nicer about this, we could also detect maybe 2 or 3 classes

Re: [DISCUSS] KIP-140: Add administrative RPCs for adding, deleting, and listing ACLs

2017-05-04 Thread dan

what about the configs for: `allow.everyone.if.no.acl.found` and `
super.users`?

i understand they are an implementation detail of
`SimpleAclAuthorizer` configs,
but without these its difficult to make sense of what permissions a
`ListAclsResponse`
actually represents.

maybe something for another kip.

dan

On Thu, May 4, 2017 at 2:36 PM, Colin McCabe  wrote:

> On Thu, May 4, 2017, at 13:46, Magnus Edenhill wrote:
> > Hey Colin,
> >
> > good KIP!
> >
> > Some comments:
> >
> > 1a. For operation, permission_type and resource_type: is there any reason
> > for having the any and unknown enums as negative values?
> > Since neither of these fields has an integer significance (unlike for
> > example offsets which use negative offsets for logical offsets) I dont
> > really see a reason to do this. It might also trick client developers to
> > make assumptions on future negative values (if resource_type < 0:
> > treat_as_invalid()...), unless that's the reason :). This might be my
> > personal preference but encoding extended meaning into types should be
> > avoided unless needed, and I dont think it is needed for enums.
>
> Hi Magnus,
>
> That's a fair question.  I don't have a strong preference either way.
> If it is more convenient or consistent to start at 0, we can certainly
> do that.
>
> >
> > but..
> >
> > 1b. Since most clients implementing the ACL requests probably wont make
> > much use of this API themselves but rather treat it as a straight
> > pass-through between the protocol and the client's public API to the
> > application, could we save ourselves some work (current and future) to
> > make
> > the enums as nullable_strings instead of integer enums? This would cut
> > down
> > on the number of enum-to-str and vice versa conversions needed, and would
> > also make the APIs more future proof since an added resource_type (et.al
> )
> > would not need a client, or even tool, update, and the new type will not
> > show up as UNKNOWN but of its real value.
> > From a broker-side verification perspective there should really be no
> > difference since the enum values will need to be interpreted anyway.
> > So instead of int enum { unknown = -2, any = -1, deny, allow }, we have {
> > null, "deny", "allow" }.
>
> Strings use much, much more space, though.  An INT8 is one byte, whereas
> the string "clusterAction" is 13 bytes, plus a 2 byte length field (if I
> remember our serialization correctly)  A 10x or 15x RPC space penalty
> starts to really hurt, especially when you have hundreds or thousands of
> ACLs, and each one has 6 fields.
>
> >
> >
> > 2. "Each of the arguments to ListAclsRequest acts as a filter."
> > Specify if these filters are OR:ed or AND:ed.
>
> Yeah, they are ANDed.  I'll add a note.
>
> >
> > 3. "CreateAclsRequest and CreateAclsResponse"
> > What happens if a client attempts to create an ACL entry which is
> > identical
> > to one already created in the cluster?
> > Is this an error? silently ignored? resulting in duplicates?
>
> Unfortunately, we are somewhat at the mercy of the
> kafka.security.auth.Authorizer implementation here.  The default
> SimpleAclAuthorizer does not allow duplicates to be added.
> If the Authorizer doesn't de-duplicate, we can't add de-duplication
> without doing distributed locking of some sort.  I think it makes sense
> to add a new exception, DuplicateAcl, though, and specify that the
> authorizer ought to throw that exception.  Let me add a little more
> detail here.
>
> >
> > 4. "Compatibility plan"
> > "If we later add new resource types, operation types, and so forth, we
> > would like to be able to interact with them with the old AdminClient.
> > This
> > is why the AclResourceType, AclOperation, and AclPermissionType enums
> > have
> > UNKNOWN values.  If we get an INT8 which we don't know the name for, it
> > gets mapped to an UNKNOWN object."
> >
> > I'm not sure who "we" are. Is it the broker or the client?
> > (also see 1b for avoiding UNKNOWN altogether)
>
> Both the broker and the client can get "unknown" values for those enums,
> if someone sends them something they don't understand.  The broker will
> not add new ACLs that contain unknowns, nor will it delete using filters
> that have unknowns.  Similarly, attempting to list ACLs with unknowns is
> an error.  It is still possible for a client to delete an ACL that it
> doesn't understand, by using ANY fields to match it.  I guess I should
> spell out all these cases in the wiki.
>
> best,
> Colin
>
> >
> >
> > Regards,
> > Magnus
> >
> > 2017-04-21 22:27 GMT+02:00 Colin McCabe :
> >
> > > Hi all,
> > >
> > > As part of the AdminClient work, we would like to add methods for
> > > adding, deleting, and listing access control lists (ACLs).  I wrote up
> a
> > > KIP to discuss implementing requests for those operations, as well as
> > > AdminClient APIs.  Take a look at:
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > >

[jira] [Commented] (KAFKA-4764) Improve diagnostics for SASL authentication failures

2017-05-04 Thread Jun Rao (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997510#comment-15997510
 ] 

Jun Rao commented on KAFKA-4764:


[~rsivaram], thanks for the explanation. I looked at KIP-152. It looks 
reasonable other than that it seems a bit overkill for just error handling. 
Ideally, SASL library should be able to exchange some error tokens natively. 
Should we look a bit more there?

> Improve diagnostics for SASL authentication failures
> 
>
> Key: KAFKA-4764
> URL: https://issues.apache.org/jira/browse/KAFKA-4764
> Project: Kafka
>  Issue Type: Improvement
>  Components: security
>Affects Versions: 0.10.2.0
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
> Fix For: 0.11.0.0
>
>
> At the moment, broker closes the client connection if SASL authentication 
> fails. Clients see this as a connection failure and do not get any feedback 
> for the reason why the connection was closed. Producers and consumers retry, 
> attempting to create successful connections, treating authentication failures 
> as transient failures. There are no log entries on the client-side which 
> indicate that any of these connection failures were due to authentication 
> failure.
> This JIRA will aim to improve diagnosis of authentication failures with the 
> following changes:
> - Broker will send an authentication error code if SASL authentication fails, 
> just before closing the connection. This will be treated as an invalid token 
> by the client authenticator, and the error handling for invalid tokens will 
> be updated to report authentication failure for this case. This is a bit of a 
> hack, but would work with GSSAPI, PLAIN and SCRAM. SASL itself doesn't 
> provide a mechanism-independent way of reporting authentication failures. An 
> alternative would be to wrap SASL authentication in Kafka request/response to 
> enables error codes to be sent as Kafka response, but that would be a much 
> bigger change.
> - Log a warning in clients for authentication failures, distinguishing these 
> from EOF exceptions due to connection failure
> - Blackout nodes to which connection failed due to authentication error, no 
> more attempts will be made to connect to these nodes.
> - We should use the connection state to improve handling of producer/consumer 
> requests, avoiding unnecessary blocking. This will not be addressed in this 
> JIRA, KAFKA-3899 should be able to use the additional state from JIRA to fix 
> this issue.
> This JIRA also does not change handling of SSL authentication failures. 
> javax.net.debug provides sufficient diagnostics for this case, I don't 
> believe there is sufficient information in `SslTransportLayer` to treat these 
> in a consistent way with SASL authentication failures.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

One follow up. There was this email on the user list:

http://search-hadoop.com/m/Kafka/uyzND17KhCaBzPSZ1?subj=Shouldn+t+the+initializer+of+a+stream+aggregate+accept+the+key+

It might make sense so include Initializer, Adder, and Substractor
inferface, too.

And we should double check if there are other interface we might miss atm.


-Matthias


On 5/4/17 1:31 PM, Matthias J. Sax wrote:
> Thanks for updating the KIP.
> 
> Deep copying the key will work for sure, but I am actually a little bit
> worried about performance impact... We might want to do some test to
> quantify this impact.
> 
> 
> Btw: this remind me about the idea of `RichFunction` interface that
> would allow users to access record metadata (like timestamp, offset,
> partition etc) within DSL. This would be a similar concept. Thus, I am
> wondering, if it would make sense to enlarge the scope of this KIP by
> that? WDYT?
> 
> 
> 
> -Matthias
> 
> 
> On 5/3/17 2:08 AM, Jeyhun Karimov wrote:
>> Hi Mathieu,
>>
>> Thanks for feedback. I followed similar approach and updated PR and KIP
>> accordingly. I tried to guard the key in Processors sending a copy of an
>> actual key.
>> Because I am doing deep copy of an object, I think memory can be bottleneck
>> in some use-cases.
>>
>> Cheers,
>> Jeyhun
>>
>> On Tue, May 2, 2017 at 5:10 PM Mathieu Fenniak 
>> wrote:
>>
>>> Hi Jeyhun,
>>>
>>> This approach would change ValueMapper (...etc) to be classes, rather than
>>> interfaces, which is also a backwards incompatible change.  An alternative
>>> approach that would be backwards compatible would be to define new
>>> interfaces, and provide overrides where those interfaces are used.
>>>
>>> Unfortunately, making the key parameter as "final" doesn't change much
>>> about guarding against key change.  It only prevents the parameter variable
>>> from being reassigned.  If the key type is a mutable object (eg. byte[]),
>>> it can still be mutated. (eg. key[0] = 0).  But I'm not really sure there's
>>> much that can be done about that.
>>>
>>> Mathieu
>>>
>>>
>>> On Mon, May 1, 2017 at 5:39 PM, Jeyhun Karimov 
>>> wrote:
>>>
 Thanks for comments.

 The concerns makes sense. Although we can guard for immutable keys in
 current implementation (with few changes), I didn't consider backward
 compatibility.

 In this case 2 solutions come to my mind. In both cases, user accesses
>>> the
 key in Object type, as passing extra type parameter will break
 backwards-compatibility.  So user has to cast to actual key type.

 1. Firstly, We can overload apply method with 2 argument (key and value)
 and force key to be *final*. By doing this,  I think we can address both
 backward-compatibility and guarding against key change.

 2. Secondly, we can create class KeyAccess like:

 public class KeyAccess {
 Object key;
 public void beforeApply(final Object key) {
 this.key = key;
 }
 public Object getKey() {
 return key;
 }
 }

 We can extend *ValueMapper, ValueJoiner* and *ValueTransformer* from
 *KeyAccess*. Inside processor (for example *KTableMapValuesProcessor*)
 before calling *mapper.apply(value)* we can set the *key* by
 *mapper.beforeApply(key)*. As a result, user can use *getKey()* to access
 the key inside *apply(value)* method.


 Cheers,
 Jeyhun




 On Mon, May 1, 2017 at 7:24 PM Matthias J. Sax 
 wrote:

> Jeyhun,
>
> thanks a lot for the KIP!
>
> I think there are two issues we need to address:
>
> (1) The KIP does not consider backward compatibility. Users did
>>> complain
> about this in past releases already, and as the user base grows, we
> should not break backward compatibility in future releases anymore.
> Thus, we should think of a better way to allow key access.
>
> Mathieu's comment goes into the same direction
>
>>> On the other hand, the number of compile failures that would need to
 be
>>> fixed from this change is unfortunate. :-)
>
> (2) Another concern is, that there is no guard to prevent user code to
> modify the key. This might corrupt partitioning if users do alter the
> key (accidentally -- or users are just not aware that they are not
> allowed to modify the provided key object) and thus break the
> application. (This was the original motivation to not provide the key
>>> in
> the first place -- it's guards against modification.)
>
>
> -Matthias
>
>
>
> On 5/1/17 6:31 AM, Mathieu Fenniak wrote:
>> Hi Jeyhun,
>>
>> I just want to add my voice that, I too, have wished for access to
>>> the
>> record key during a mapValues or similar operation.
>>
>> On the other hand, the number of compile failures that would need to
>>> be

[jira] [Assigned] (KAFKA-5175) Transient failure: ControllerIntegrationTest.testPreferredReplicaLeaderElection

2017-05-04 Thread Onur Karaman (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-5175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Onur Karaman reassigned KAFKA-5175:
---

Assignee: Onur Karaman

> Transient failure: 
> ControllerIntegrationTest.testPreferredReplicaLeaderElection
> ---
>
> Key: KAFKA-5175
> URL: https://issues.apache.org/jira/browse/KAFKA-5175
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Ismael Juma
>Assignee: Onur Karaman
>
> {code}
> java.lang.AssertionError: failed to get expected partition state upon broker 
> startup
>   at kafka.utils.TestUtils$.fail(TestUtils.scala:311)
>   at kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:811)
>   at 
> kafka.controller.ControllerIntegrationTest.waitForPartitionState(ControllerIntegrationTest.scala:293)
>   at 
> kafka.controller.ControllerIntegrationTest.testPreferredReplicaLeaderElection(ControllerIntegrationTest.scala:211)
> {code}
> https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/3497/testReport/kafka.controller/ControllerIntegrationTest/testPreferredReplicaLeaderElection/



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (KAFKA-5175) Transient failure: ControllerIntegrationTest.testPreferredReplicaLeaderElection

Ismael Juma created KAFKA-5175:
--

 Summary: Transient failure: 
ControllerIntegrationTest.testPreferredReplicaLeaderElection
 Key: KAFKA-5175
 URL: https://issues.apache.org/jira/browse/KAFKA-5175
 Project: Kafka
  Issue Type: Sub-task
Reporter: Ismael Juma


{code}
java.lang.AssertionError: failed to get expected partition state upon broker 
startup
at kafka.utils.TestUtils$.fail(TestUtils.scala:311)
at kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:811)
at 
kafka.controller.ControllerIntegrationTest.waitForPartitionState(ControllerIntegrationTest.scala:293)
at 
kafka.controller.ControllerIntegrationTest.testPreferredReplicaLeaderElection(ControllerIntegrationTest.scala:211)
{code}

https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/3497/testReport/kafka.controller/ControllerIntegrationTest/testPreferredReplicaLeaderElection/



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5175) Transient failure: ControllerIntegrationTest.testPreferredReplicaLeaderElection


[ 
https://issues.apache.org/jira/browse/KAFKA-5175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997497#comment-15997497
 ] 

Ismael Juma commented on KAFKA-5175:


[~onurkaraman], can you please take a look?

> Transient failure: 
> ControllerIntegrationTest.testPreferredReplicaLeaderElection
> ---
>
> Key: KAFKA-5175
> URL: https://issues.apache.org/jira/browse/KAFKA-5175
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Ismael Juma
>
> {code}
> java.lang.AssertionError: failed to get expected partition state upon broker 
> startup
>   at kafka.utils.TestUtils$.fail(TestUtils.scala:311)
>   at kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:811)
>   at 
> kafka.controller.ControllerIntegrationTest.waitForPartitionState(ControllerIntegrationTest.scala:293)
>   at 
> kafka.controller.ControllerIntegrationTest.testPreferredReplicaLeaderElection(ControllerIntegrationTest.scala:211)
> {code}
> https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/3497/testReport/kafka.controller/ControllerIntegrationTest/testPreferredReplicaLeaderElection/



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [DISCUSS] KIP-140: Add administrative RPCs for adding, deleting, and listing ACLs

2017-05-04 Thread Colin McCabe

On Thu, May 4, 2017, at 13:46, Magnus Edenhill wrote:
> Hey Colin,
> 
> good KIP!
> 
> Some comments:
> 
> 1a. For operation, permission_type and resource_type: is there any reason
> for having the any and unknown enums as negative values?
> Since neither of these fields has an integer significance (unlike for
> example offsets which use negative offsets for logical offsets) I dont
> really see a reason to do this. It might also trick client developers to
> make assumptions on future negative values (if resource_type < 0:
> treat_as_invalid()...), unless that's the reason :). This might be my
> personal preference but encoding extended meaning into types should be
> avoided unless needed, and I dont think it is needed for enums.

Hi Magnus,

That's a fair question.  I don't have a strong preference either way. 
If it is more convenient or consistent to start at 0, we can certainly
do that.

> 
> but..
> 
> 1b. Since most clients implementing the ACL requests probably wont make
> much use of this API themselves but rather treat it as a straight
> pass-through between the protocol and the client's public API to the
> application, could we save ourselves some work (current and future) to
> make
> the enums as nullable_strings instead of integer enums? This would cut
> down
> on the number of enum-to-str and vice versa conversions needed, and would
> also make the APIs more future proof since an added resource_type (et.al)
> would not need a client, or even tool, update, and the new type will not
> show up as UNKNOWN but of its real value.
> From a broker-side verification perspective there should really be no
> difference since the enum values will need to be interpreted anyway.
> So instead of int enum { unknown = -2, any = -1, deny, allow }, we have {
> null, "deny", "allow" }.

Strings use much, much more space, though.  An INT8 is one byte, whereas
the string "clusterAction" is 13 bytes, plus a 2 byte length field (if I
remember our serialization correctly)  A 10x or 15x RPC space penalty
starts to really hurt, especially when you have hundreds or thousands of
ACLs, and each one has 6 fields.

> 
> 
> 2. "Each of the arguments to ListAclsRequest acts as a filter."
> Specify if these filters are OR:ed or AND:ed.

Yeah, they are ANDed.  I'll add a note.

> 
> 3. "CreateAclsRequest and CreateAclsResponse"
> What happens if a client attempts to create an ACL entry which is
> identical
> to one already created in the cluster?
> Is this an error? silently ignored? resulting in duplicates?

Unfortunately, we are somewhat at the mercy of the
kafka.security.auth.Authorizer implementation here.  The default
SimpleAclAuthorizer does not allow duplicates to be added.
If the Authorizer doesn't de-duplicate, we can't add de-duplication
without doing distributed locking of some sort.  I think it makes sense
to add a new exception, DuplicateAcl, though, and specify that the
authorizer ought to throw that exception.  Let me add a little more
detail here.

> 
> 4. "Compatibility plan"
> "If we later add new resource types, operation types, and so forth, we
> would like to be able to interact with them with the old AdminClient. 
> This
> is why the AclResourceType, AclOperation, and AclPermissionType enums
> have
> UNKNOWN values.  If we get an INT8 which we don't know the name for, it
> gets mapped to an UNKNOWN object."
> 
> I'm not sure who "we" are. Is it the broker or the client?
> (also see 1b for avoiding UNKNOWN altogether)

Both the broker and the client can get "unknown" values for those enums,
if someone sends them something they don't understand.  The broker will
not add new ACLs that contain unknowns, nor will it delete using filters
that have unknowns.  Similarly, attempting to list ACLs with unknowns is
an error.  It is still possible for a client to delete an ACL that it
doesn't understand, by using ANY fields to match it.  I guess I should
spell out all these cases in the wiki.

best,
Colin

> 
> 
> Regards,
> Magnus
> 
> 2017-04-21 22:27 GMT+02:00 Colin McCabe :
> 
> > Hi all,
> >
> > As part of the AdminClient work, we would like to add methods for
> > adding, deleting, and listing access control lists (ACLs).  I wrote up a
> > KIP to discuss implementing requests for those operations, as well as
> > AdminClient APIs.  Take a look at:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 140%3A+Add+administrative+RPCs+for+adding%2C+deleting%2C+and+listing+ACLs
> >
> > regards,
> > Colin
> >

[jira] [Commented] (KAFKA-5166) Add option "dry run" to Streams application reset tool


[ 
https://issues.apache.org/jira/browse/KAFKA-5166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997464#comment-15997464
 ] 

Bharat Viswanadham commented on KAFKA-5166:
---

Hi Matthias,
I have not written KIPS. I will look in to it and provide KIP.
But i have a question why public API change will happen.
My idea is to add a new option --dry-run and print the info to user, what all 
actions will be performed.

Could you please let me know anything I am missing here.

> Add option "dry run" to Streams application reset tool
> --
>
> Key: KAFKA-5166
> URL: https://issues.apache.org/jira/browse/KAFKA-5166
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams, tools
>Affects Versions: 0.10.2.0
>Reporter: Matthias J. Sax
>Assignee: Bharat Viswanadham
>Priority: Minor
>  Labels: needs-kip
>
> We want to add an option to Streams application reset tool, that allow for a 
> "dry run". Ie, only prints what topics would get modified/deleted without 
> actually applying any actions.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [DISCUSS] KIP-152 - Improve diagnostics for SASL authentication failures

Hi Rajini,

I think we were talking about slightly different things. I was just
referring to the fact that there are cases where we throw an
AuthorizationException back to the user without retrying from various
methods (poll, commitSync, etc).

As you said, my initial preference was for not retrying at all because it
is what you want in the common case of a misconfigured application. I
hadn't considered credential updates for authenticators that rely on
eventual consistency. Thinking about it some more, it seems like this
should be solved by the authenticator implementation as well. For example,
it could refresh the cached data for a user if authentication failed (a
good implementation would be a bit more involved to avoid going to the
underlying data source too often).

Given that, not retrying sounds good to me.

Ismael

On Thu, May 4, 2017 at 4:04 PM, Rajini Sivaram 
wrote:

> Hi Ismael,
>
> I thought the blocking waits in the producer and consumer are always
> related to retrying for metadata. So an authorization exception that
> impacts this wait can only be due to Describe authorization failure - that
> always retries?
>
> I agree that connecting to different brokers when authentication fails with
> one is not desirable. But I am not keen on retrying with a suitable backoff
> until timeout either. Because that has the same problem as the scenario
> that you described. The next metadata request could be to broker-1 to which
> authentication succeeds and subsequent produce/consume  to broker-0 could
> still fail.
>
> How about we just fail fast if one authentication fails - I think that is
> what you were suggesting in the first place? We don't need to blackout any
> nodes beyond the reconnect backoff interval. Applications can still retry
> if they want to. In the case of credential updates, it will be up to the
> application to retry. During regular operation, a misconfigured application
> fails fast with a meaningful exception. What do you think?
>
> Regards,
>
> Rajini
>
>
> On Thu, May 4, 2017 at 3:01 PM, Ismael Juma  wrote:
>
> > H Rajini,
> >
> > Comments inline.
> >
> > On Thu, May 4, 2017 at 2:29 PM, Rajini Sivaram 
> > wrote:
> >
> > > Hi Ismael,
> > >
> > > Thank you for reviewing the KIP.
> > >
> > > An authenticated client that is not authorized to access a topic is
> never
> > > told that the operation was not authorized. This is to prevent the
> client
> > > from finding out if the topic exists by sending an unauthorized
> request.
> > So
> > > in this case, the client will retry metadata requests with the
> configured
> > > backoff until it times out.
> >
> >
> > This is true if the user does not have Describe permission. If the user
> has
> > Describe access and no Read or Write access, then the user is informed
> that
> > the operation was not authorized.
> >
> >
> > > Another important distinction for authorization failures is that the
> > > connection is not terminated.
> > >
> > > For unauthenticated clients, we do want to inform the client that
> > > authentication failed. The connection is terminated by the broker.
> > > Especially if the client is using SASL_SSL, we really do want to avoid
> > > reconnections that result in unnecessary expensive handshakes. So we
> want
> > > to return an exception to the user with minimal retries.
> > >
> >
> > Agreed.
> >
> > I was thinking that it may be useful to try more than one broker for the
> > > case where brokers are being upgraded and some brokers haven't yet seen
> > the
> > > latest credentials. I suppose I was thinking that at the moment we keep
> > on
> > > retrying every broker forever in the consumer and suddenly if we stop
> > > retrying altogether, it could potentially lead to some unforeseen
> timing
> > > issues. Hence the suggestion to try every broker once.
> > >
> >
> > I see. Retrying forever is a side-effect of auto-topic creation, but it's
> > something we want to move away from. As mentioned, we actually don't
> retry
> > at all if the user has Describe permission.
> >
> > Broker upgrades could be fixed by ensuring that the latest credentials
> are
> > loaded before the broker starts serving requests. More problematic is
> > dealing with credential updates. This is another distinction when
> compared
> > to authorization.
> >
> > I am not sure if trying different brokers really helps us though. Say, we
> > fail to authenticate with broker 0 and then we succeed with broker 1.
> This
> > helps with metadata requests, but we will be in trouble when we try to
> > produce or consume to broker 0 (because it's the leader of some
> > partitions). So maybe we just want to retry with a suitable backoff
> until a
> > timeout?
> >
> > Yes, I agree that blacking out nodes forever isn't a good idea. When we
> > > throw AuthenticationFailedException for the current operation or if
> > > authentication to another broker succeeds, we can clear the blackout so
> > >

Build failed in Jenkins: kafka-trunk-jdk7 #2149

See 


Changes:

[wangguoz] KAFKA-5118: Improve message for Kafka failed startup with non-Kafka 
data

[wangguoz] KAFKA-5104: DumpLogSegments should not open index files with `rw`

--
[...truncated 843.38 KB...]
kafka.tools.ConsoleConsumerTest > shouldLimitReadsToMaxMessageLimit STARTED

kafka.tools.ConsoleConsumerTest > shouldLimitReadsToMaxMessageLimit PASSED

kafka.tools.ConsoleConsumerTest > shouldParseValidNewConsumerValidConfig STARTED

kafka.tools.ConsoleConsumerTest > shouldParseValidNewConsumerValidConfig PASSED

kafka.tools.ConsoleConsumerTest > shouldStopWhenOutputCheckErrorFails STARTED

kafka.tools.ConsoleConsumerTest > shouldStopWhenOutputCheckErrorFails PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewSimpleConsumerValidConfigWithStringOffset STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewSimpleConsumerValidConfigWithStringOffset PASSED

kafka.tools.ConsoleConsumerTest > shouldParseConfigsFromFile STARTED

kafka.tools.ConsoleConsumerTest > shouldParseConfigsFromFile PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewSimpleConsumerValidConfigWithNumericOffset STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewSimpleConsumerValidConfigWithNumericOffset PASSED

kafka.tools.ConsoleConsumerTest > testDefaultConsumer STARTED

kafka.tools.ConsoleConsumerTest > testDefaultConsumer PASSED

kafka.tools.ConsoleConsumerTest > shouldParseValidOldConsumerValidConfig STARTED

kafka.tools.ConsoleConsumerTest > shouldParseValidOldConsumerValidConfig PASSED

kafka.tools.ConsoleProducerTest > testParseKeyProp STARTED

kafka.tools.ConsoleProducerTest > testParseKeyProp PASSED

kafka.tools.ConsoleProducerTest > testValidConfigsOldProducer STARTED

kafka.tools.ConsoleProducerTest > testValidConfigsOldProducer PASSED

kafka.tools.ConsoleProducerTest > testInvalidConfigs STARTED

kafka.tools.ConsoleProducerTest > testInvalidConfigs PASSED

kafka.tools.ConsoleProducerTest > testValidConfigsNewProducer STARTED

kafka.tools.ConsoleProducerTest > testValidConfigsNewProducer PASSED

kafka.tools.MirrorMakerTest > 
testDefaultMirrorMakerMessageHandlerWithNoTimestampInSourceMessage STARTED

kafka.tools.MirrorMakerTest > 
testDefaultMirrorMakerMessageHandlerWithNoTimestampInSourceMessage PASSED

kafka.tools.MirrorMakerTest > testDefaultMirrorMakerMessageHandler STARTED

kafka.tools.MirrorMakerTest > testDefaultMirrorMakerMessageHandler PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytesWithCompression 
STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytesWithCompression 
PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > 
testIteratorIsConsistentWithCompression STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > 
testIteratorIsConsistentWithCompression PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testIteratorIsConsistent 
STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testIteratorIsConsistent PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testEqualsWithCompression 
STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testEqualsWithCompression 
PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testWrittenEqualsRead STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testWrittenEqualsRead PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testEquals STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testEquals PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytes STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytes PASSED

kafka.javaapi.consumer.ZookeeperConsumerConnectorTest > testBasic STARTED

kafka.javaapi.consumer.ZookeeperConsumerConnectorTest > testBasic PASSED

kafka.common.InterBrokerSendThreadTest > 
shouldCreateClientRequestAndSendWhenNodeIsReady STARTED

kafka.common.InterBrokerSendThreadTest > 
shouldCreateClientRequestAndSendWhenNodeIsReady PASSED

kafka.common.InterBrokerSendThreadTest > 
shouldCallCompletionHandlerWithDisconnectedResponseWhenNodeNotReady STARTED

kafka.common.InterBrokerSendThreadTest > 
shouldCallCompletionHandlerWithDisconnectedResponseWhenNodeNotReady PASSED

kafka.common.InterBrokerSendThreadTest > shouldNotSendAnythingWhenNoRequests 
STARTED

kafka.common.InterBrokerSendThreadTest > shouldNotSendAnythingWhenNoRequests 
PASSED

kafka.common.ConfigTest > testInvalidGroupIds STARTED

kafka.common.ConfigTest > testInvalidGroupIds PASSED

kafka.common.ConfigTest > testInvalidClientIds STARTED

kafka.common.ConfigTest > testInvalidClientIds PASSED

kafka.common.ZkNodeChangeNotificationListenerTest > testProcessNotification 
STARTED

kafka.common.ZkNodeChangeNotificationListenerTest > testProcessNotification 
PASSED

kafka.common.TopicTest > testInvalidTopicNames STARTED

kafka.common.TopicTest > testInvalidTopicNames PASSED

kafka.common.TopicTest >

[jira] [Commented] (KAFKA-5166) Add option "dry run" to Streams application reset tool


[ 
https://issues.apache.org/jira/browse/KAFKA-5166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997427#comment-15997427
 ] 

Matthias J. Sax commented on KAFKA-5166:


Thanks for picking this up [~bharatviswa]! Are you familiar with KIP process: 
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals

Should not be hard to get done for this ticket, but it's requires as is changes 
public API.

> Add option "dry run" to Streams application reset tool
> --
>
> Key: KAFKA-5166
> URL: https://issues.apache.org/jira/browse/KAFKA-5166
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams, tools
>Affects Versions: 0.10.2.0
>Reporter: Matthias J. Sax
>Assignee: Bharat Viswanadham
>Priority: Minor
>  Labels: needs-kip
>
> We want to add an option to Streams application reset tool, that allow for a 
> "dry run". Ie, only prints what topics would get modified/deleted without 
> actually applying any actions.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5166) Add option "dry run" to Streams application reset tool


[ 
https://issues.apache.org/jira/browse/KAFKA-5166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997426#comment-15997426
 ] 

Bharat Viswanadham commented on KAFKA-5166:
---

Hi Matthias,
I will look into this, and come up with a proposal.

> Add option "dry run" to Streams application reset tool
> --
>
> Key: KAFKA-5166
> URL: https://issues.apache.org/jira/browse/KAFKA-5166
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams, tools
>Affects Versions: 0.10.2.0
>Reporter: Matthias J. Sax
>Assignee: Bharat Viswanadham
>Priority: Minor
>  Labels: needs-kip
>
> We want to add an option to Streams application reset tool, that allow for a 
> "dry run". Ie, only prints what topics would get modified/deleted without 
> actually applying any actions.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (KAFKA-5166) Add option "dry run" to Streams application reset tool


 [ 
https://issues.apache.org/jira/browse/KAFKA-5166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bharat Viswanadham reassigned KAFKA-5166:
-

Assignee: Bharat Viswanadham

> Add option "dry run" to Streams application reset tool
> --
>
> Key: KAFKA-5166
> URL: https://issues.apache.org/jira/browse/KAFKA-5166
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams, tools
>Affects Versions: 0.10.2.0
>Reporter: Matthias J. Sax
>Assignee: Bharat Viswanadham
>Priority: Minor
>  Labels: needs-kip
>
> We want to add an option to Streams application reset tool, that allow for a 
> "dry run". Ie, only prints what topics would get modified/deleted without 
> actually applying any actions.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4764) Improve diagnostics for SASL authentication failures

2017-05-04 Thread Rajini Sivaram (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997421#comment-15997421
 ] 

Rajini Sivaram commented on KAFKA-4764:
---

[~junrao] Yes, Zookeeper does the same as Kafka does today. As an example, with 
a wrong password for the ZK client, the logs in Kafka show:
{quote}
DEBUG saslClient.evaluateChallenge(len=101) 
(org.apache.zookeeper.client.ZooKeeperSaslClient)
DEBUG ClientCnxn:sendSaslPacket:length=276 
(org.apache.zookeeper.client.ZooKeeperSaslClient)
INFO Unable to read additional data from server sessionid 0x15bd52108ef0003, 
likely server has closed socket, closing socket connection and attempting 
reconnect (org.apache.zookeeper.ClientCnxn)
{quote}

The issue with treating any disconnects during authentication as authentication 
failures is that we may incorrectly report authentication failure for a 
correctly configured client because a broker was restarting. Also, if we want 
to treat authentication exceptions as non-retriable exceptions, we need to be 
sure that the connection failure was actually an authentication exception. I 
suppose if we want the simplest solution, we could simply log that connection 
failed during authentication with a hint that it may be an authentication 
failure. And retry as we do today.

The proposal in 
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-152+-+Improve+diagnostics+for+SASL+authentication+failures]
 is different from the description in this JIRA. In case you haven't seen the 
KIP, can you take a look? It is a much bigger change, but helps to ensure that 
we can identify and treat authentication failures differently from any other 
form of network error. 

Thank you

> Improve diagnostics for SASL authentication failures
> 
>
> Key: KAFKA-4764
> URL: https://issues.apache.org/jira/browse/KAFKA-4764
> Project: Kafka
>  Issue Type: Improvement
>  Components: security
>Affects Versions: 0.10.2.0
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
> Fix For: 0.11.0.0
>
>
> At the moment, broker closes the client connection if SASL authentication 
> fails. Clients see this as a connection failure and do not get any feedback 
> for the reason why the connection was closed. Producers and consumers retry, 
> attempting to create successful connections, treating authentication failures 
> as transient failures. There are no log entries on the client-side which 
> indicate that any of these connection failures were due to authentication 
> failure.
> This JIRA will aim to improve diagnosis of authentication failures with the 
> following changes:
> - Broker will send an authentication error code if SASL authentication fails, 
> just before closing the connection. This will be treated as an invalid token 
> by the client authenticator, and the error handling for invalid tokens will 
> be updated to report authentication failure for this case. This is a bit of a 
> hack, but would work with GSSAPI, PLAIN and SCRAM. SASL itself doesn't 
> provide a mechanism-independent way of reporting authentication failures. An 
> alternative would be to wrap SASL authentication in Kafka request/response to 
> enables error codes to be sent as Kafka response, but that would be a much 
> bigger change.
> - Log a warning in clients for authentication failures, distinguishing these 
> from EOF exceptions due to connection failure
> - Blackout nodes to which connection failed due to authentication error, no 
> more attempts will be made to connect to these nodes.
> - We should use the connection state to improve handling of producer/consumer 
> requests, avoiding unnecessary blocking. This will not be addressed in this 
> JIRA, KAFKA-3899 should be able to use the additional state from JIRA to fix 
> this issue.
> This JIRA also does not change handling of SSL authentication failures. 
> javax.net.debug provides sufficient diagnostics for this case, I don't 
> believe there is sufficient information in `SslTransportLayer` to treat these 
> in a consistent way with SASL authentication failures.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [DISCUSS] KIP-152 - Improve diagnostics for SASL authentication failures

2017-05-04 Thread Magnus Edenhill

Hi Rajini, great KIP!
This solution was proposed on the original KIP-43 thread but voted down, so
let's hope it does better this time :)

/Magnus


2017-05-04 13:37 GMT+02:00 Rajini Sivaram :

> Hi all,
>
> I have created a KIP to improve diagnostics for SASL authentication
> failures and reduce retries and blocking when authentication fails:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 152+-+Improve+diagnostics+for+SASL+authentication+failures
>
> Comments and suggestions are welcome.
>
> Thank you...
>
> Regards,
>
> Rajini
>

Re: [DISCUSS] KIP-147: Add missing type parameters to StateStoreSupplier factories and KGroupedStream/Table methods

I had a quick look into this.

With regard to backward compatibility, I think it would be required do
introduce a new type `TypesStateStoreSupplier` (that extends
`StateStoreSupplier`) and to overload all methods that take a
`StateStoreSupplier` that accept the new type instead of the current one.

This would allow `.build` to return a `TypedStateStoreSupplier` and
thus, would not break any code. As least if I did not miss anything with
regard to some magic of type inference using generics (I am not an
expert in this field).


-Matthias

On 5/4/17 11:32 AM, Matthias J. Sax wrote:
> Did not have time to have a look. But backward compatibility is a must
> from my point of view.
> 
> -Matthias
> 
> 
> On 5/4/17 12:56 AM, Michal Borowiecki wrote:
>> Hello,
>>
>> I've updated the KIP with missing information.
>>
>> I would especially appreciate some comments on the compatibility aspects
>> of this as the proposed change is not fully backwards-compatible.
>>
>> In the absence of comments I shall call for a vote in the next few days.
>>
>> Thanks,
>>
>> Michal
>>
>>
>> On 30/04/17 23:11, Michal Borowiecki wrote:
>>>
>>> Hi community!
>>>
>>> I have just drafted KIP-147: Add missing type parameters to
>>> StateStoreSupplier factories and KGroupedStream/Table methods
>>> 
>>>
>>> Please let me know if this a step in the right direction.
>>>
>>> All comments welcome.
>>>
>>> Thanks,
>>> Michal
>>> -- 
>>> Signature
>>>    Michal Borowiecki
>>> Senior Software Engineer L4
>>> T:  +44 208 742 1600
>>>
>>> 
>>> +44 203 249 8448
>>>
>>> 
>>>  
>>> E:  michal.borowie...@openbet.com
>>> W:  www.openbet.com 
>>>
>>> 
>>> OpenBet Ltd
>>>
>>> Chiswick Park Building 9
>>>
>>> 566 Chiswick High Rd
>>>
>>> London
>>>
>>> W4 5XT
>>>
>>> UK
>>>
>>> 
>>> 
>>>
>>> This message is confidential and intended only for the addressee. If
>>> you have received this message in error, please immediately notify the
>>> postmas...@openbet.com  and delete it
>>> from your system as well as any copies. The content of e-mails as well
>>> as traffic data may be monitored by OpenBet for employment and
>>> security purposes. To protect the environment please do not print this
>>> e-mail unless necessary. OpenBet Ltd. Registered Office: Chiswick Park
>>> Building 9, 566 Chiswick High Road, London, W4 5XT, United Kingdom. A
>>> company registered in England and Wales. Registered no. 3134634. VAT
>>> no. GB927523612
>>>
>>
>> -- 
>> Signature
>> Michal Borowiecki
>> Senior Software Engineer L4
>>  T:  +44 208 742 1600
>>
>>  
>>  +44 203 249 8448
>>
>>  
>>   
>>  E:  michal.borowie...@openbet.com
>>  W:  www.openbet.com 
>>
>>  
>>  OpenBet Ltd
>>
>>  Chiswick Park Building 9
>>
>>  566 Chiswick High Rd
>>
>>  London
>>
>>  W4 5XT
>>
>>  UK
>>
>>  
>> 
>>
>> This message is confidential and intended only for the addressee. If you
>> have received this message in error, please immediately notify the
>> postmas...@openbet.com  and delete it
>> from your system as well as any copies. The content of e-mails as well
>> as traffic data may be monitored by OpenBet for employment and security
>> purposes. To protect the environment please do not print this e-mail
>> unless necessary. OpenBet Ltd. Registered Office: Chiswick Park Building
>> 9, 566 Chiswick High Road, London, W4 5XT, United Kingdom. A company
>> registered in England and Wales. Registered no. 3134634. VAT no.
>> GB927523612
>>
> 



signature.asc
Description: OpenPGP digital signature

Re: [VOTE] KIP-140: Add administrative RPCs for adding, deleting, and listing ACLs

2017-05-04 Thread Colin McCabe

Hi Guozhang,

Thanks for taking a look.

On Thu, May 4, 2017, at 00:43, Guozhang Wang wrote:
> Colin,
> 
> Thanks for the KIP. A general question I have is whether we can support
> "wildcard" resource names as well? For example, if we want to create /
> delete ACLs for all topic names following a wildcard regex? I read though
> the principle wildcards description but I am not sure if it is designed
> for
> this purposes.

If you want to delete all ACLs that apply to topics, you could use a
deletion filter with resourceType=AclResourceType.TOPIC,
resourceName=null.  I think that answers the question, unless I
misunderstood.

Note that it's important to distinguish between wildcards (the "*"
value) and "any".  For example, when your deletionFilter has
principal="User:*" you are not deleting ACLs for all users.  Just ACLs
that have the principal field set to "User:*"  If you want to delete
ACLs for all users, you can use principal=null.  Similarly,
principal=null is only valid in a filter, not in an actual ACL that is
applied to a resource.

best,
Colin

> 
> Otherwise, lgtm!
> 
> 
> Guozhang
> 
> 
> On Fri, Apr 28, 2017 at 10:37 PM, Dongjin Lee  wrote:
> 
> > +1
> >
> > On 29 Apr 2017, 9:51 AM +0900, Michael Pearce ,
> > wrote:
> > > +1
> > > 
> > > From: Colin McCabe  > > Sent: Saturday, April 29, 2017 1:09:25 AM
> > > To: dev@kafka.apache.org
> > > Subject: [VOTE] KIP-140: Add administrative RPCs for adding, deleting,
> > and listing ACLs
> > >
> > > Hi all,
> > >
> > > I'd like to start the voting for KIP-140: Add administrative RPCs for
> > > adding, deleting, and listing ACLs. This provides an API for adding,
> > > deleting, and listing the access control lists (ACLs) which are used to
> > > control access on Kafka topics and brokers.
> > >
> > > The wiki page is here:
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 140%3A+Add+administrative+RPCs+for+adding%2C+deleting%2C+and+listing+ACLs
> > >
> > > The previous [DISCUSS] thread:
> > > https://www.mail-archive.com/dev@kafka.apache.org/msg70858.html
> > >
> > > cheers,
> > > Colin
> > > The information contained in this email is strictly confidential and for
> > the use of the addressee only, unless otherwise indicated. If you are not
> > the intended recipient, please do not read, copy, use or disclose to others
> > this message or any attachment. Please also notify the sender by replying
> > to this email or by telephone (+44(020 7896 0011 (tel:020%207896%200011))
> > and then delete the email and any copies of it. Opinions, conclusion (etc)
> > that do not relate to the official business of this company shall be
> > understood as neither given nor endorsed by it. IG is a trading name of IG
> > Markets Limited (a company registered in England and Wales, company number
> > 04008957 (tel:04008957)) and IG Index Limited (a company registered in
> > England and Wales, company number 01190902 (tel:01190902)). Registered
> > address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG
> > Markets Limited (register number 195355) and IG Index Limited (register
> > number 114059) are authorised and regulated
> >  by the Financial Conduct Authority.
> >
> 
> 
> 
> -- 
> -- Guozhang

Re: [DISCUSS] KIP-140: Add administrative RPCs for adding, deleting, and listing ACLs

2017-05-04 Thread Magnus Edenhill

Hey Colin,

good KIP!

Some comments:

1a. For operation, permission_type and resource_type: is there any reason
for having the any and unknown enums as negative values?
Since neither of these fields has an integer significance (unlike for
example offsets which use negative offsets for logical offsets) I dont
really see a reason to do this. It might also trick client developers to
make assumptions on future negative values (if resource_type < 0:
treat_as_invalid()...), unless that's the reason :). This might be my
personal preference but encoding extended meaning into types should be
avoided unless needed, and I dont think it is needed for enums.

but..

1b. Since most clients implementing the ACL requests probably wont make
much use of this API themselves but rather treat it as a straight
pass-through between the protocol and the client's public API to the
application, could we save ourselves some work (current and future) to make
the enums as nullable_strings instead of integer enums? This would cut down
on the number of enum-to-str and vice versa conversions needed, and would
also make the APIs more future proof since an added resource_type (et.al)
would not need a client, or even tool, update, and the new type will not
show up as UNKNOWN but of its real value.
>From a broker-side verification perspective there should really be no
difference since the enum values will need to be interpreted anyway.
So instead of int enum { unknown = -2, any = -1, deny, allow }, we have {
null, "deny", "allow" }.


2. "Each of the arguments to ListAclsRequest acts as a filter."
Specify if these filters are OR:ed or AND:ed.

3. "CreateAclsRequest and CreateAclsResponse"
What happens if a client attempts to create an ACL entry which is identical
to one already created in the cluster?
Is this an error? silently ignored? resulting in duplicates?

4. "Compatibility plan"
"If we later add new resource types, operation types, and so forth, we
would like to be able to interact with them with the old AdminClient.  This
is why the AclResourceType, AclOperation, and AclPermissionType enums have
UNKNOWN values.  If we get an INT8 which we don't know the name for, it
gets mapped to an UNKNOWN object."

I'm not sure who "we" are. Is it the broker or the client?
(also see 1b for avoiding UNKNOWN altogether)


Regards,
Magnus

2017-04-21 22:27 GMT+02:00 Colin McCabe :

> Hi all,
>
> As part of the AdminClient work, we would like to add methods for
> adding, deleting, and listing access control lists (ACLs).  I wrote up a
> KIP to discuss implementing requests for those operations, as well as
> AdminClient APIs.  Take a look at:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 140%3A+Add+administrative+RPCs+for+adding%2C+deleting%2C+and+listing+ACLs
>
> regards,
> Colin
>

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

Kyle,

thanks a lot for the KIP. Maybe I am a little slow, but I could not
follow completely. Could you maybe add a more concrete example, like 3
streams with 3 records each (plus expected result), and show the
difference between current way to to implement it and the proposed API?
This could also cover the internal processing to see what store calls
would be required for both approaches etc.

I think, it's pretty advanced stuff you propose, and it would help to
understand it better.

Thanks a lot!


-Matthias



On 5/4/17 11:39 AM, Kyle Winkelman wrote:
> I have made a pull request. It can be found here.
> 
> https://github.com/apache/kafka/pull/2975
> 
> I plan to write some more unit tests for my classes and get around to
> writing documentation for the public api additions.
> 
> One thing I was curious about is during the KCogroupedStreamImpl#aggregate
> method I pass null to the KGroupedStream#repartitionIfRequired method. I
> can't supply the store name because if more than one grouped stream
> repartitions an error is thrown. Is there some name that someone can
> recommend or should I leave the null and allow it to fall back to the
> KGroupedStream.name?
> 
> Should this be expanded to handle grouped tables? This would be pretty easy
> for a normal aggregate but one allowing session stores and windowed stores
> would required KTableSessionWindowAggregate and KTableWindowAggregate
> implementations.
> 
> Thanks,
> Kyle
> 
> On May 4, 2017 1:24 PM, "Eno Thereska"  wrote:
> 
>> I’ll look as well asap, sorry, been swamped.
>>
>> Eno
>>> On May 4, 2017, at 6:17 PM, Damian Guy  wrote:
>>>
>>> Hi Kyle,
>>>
>>> Thanks for the KIP. I apologize that i haven't had the chance to look at
>>> the KIP yet, but will schedule some time to look into it tomorrow. For
>> the
>>> implementation, can you raise a PR against kafka trunk and mark it as
>> WIP?
>>> It will be easier to review what you have done.
>>>
>>> Thanks,
>>> Damian
>>>
>>> On Thu, 4 May 2017 at 11:50 Kyle Winkelman 
>> wrote:
>>>
 I am replying to this in hopes it will draw some attention to my KIP as
>> I
 haven't heard from anyone in a couple days. This is my first KIP and my
 first large contribution to the project so I'm sure I did something
>> wrong.
 ;)

 On May 1, 2017 4:18 PM, "Kyle Winkelman" 
>> wrote:

> Hello all,
>
> I have created KIP-150 to facilitate discussion about adding cogroup to
> the streams DSL.
>
> Please find the KIP here:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 150+-+Kafka-Streams+Cogroup
>
> Please find my initial implementation here:
> https://github.com/KyleWinkelman/kafka
>
> Thanks,
> Kyle Winkelman
>

>>
>>
> 



signature.asc
Description: OpenPGP digital signature

Build failed in Jenkins: kafka-trunk-jdk8 #1482

See 


Changes:

[wangguoz] KAFKA-5118: Improve message for Kafka failed startup with non-Kafka 
data

--
[...truncated 844.67 KB...]
kafka.utils.CommandLineUtilsTest > testParseArgs STARTED

kafka.utils.CommandLineUtilsTest > testParseArgs PASSED

kafka.utils.CommandLineUtilsTest > testParseEmptyArgAsValid STARTED

kafka.utils.CommandLineUtilsTest > testParseEmptyArgAsValid PASSED

kafka.utils.ReplicationUtilsTest > testUpdateLeaderAndIsr STARTED

kafka.utils.ReplicationUtilsTest > testUpdateLeaderAndIsr PASSED

kafka.utils.ReplicationUtilsTest > testGetLeaderIsrAndEpochForPartition STARTED

kafka.utils.ReplicationUtilsTest > testGetLeaderIsrAndEpochForPartition PASSED

kafka.utils.JsonTest > testJsonEncoding STARTED

kafka.utils.JsonTest > testJsonEncoding PASSED

kafka.utils.ShutdownableThreadTest > testShutdownWhenCalledAfterThreadStart 
STARTED

kafka.utils.ShutdownableThreadTest > testShutdownWhenCalledAfterThreadStart 
PASSED

kafka.utils.SchedulerTest > testMockSchedulerNonPeriodicTask STARTED

kafka.utils.SchedulerTest > testMockSchedulerNonPeriodicTask PASSED

kafka.utils.SchedulerTest > testMockSchedulerPeriodicTask STARTED

kafka.utils.SchedulerTest > testMockSchedulerPeriodicTask PASSED

kafka.utils.SchedulerTest > testNonPeriodicTask STARTED

kafka.utils.SchedulerTest > testNonPeriodicTask PASSED

kafka.utils.SchedulerTest > testRestart STARTED

kafka.utils.SchedulerTest > testRestart PASSED

kafka.utils.SchedulerTest > testReentrantTaskInMockScheduler STARTED

kafka.utils.SchedulerTest > testReentrantTaskInMockScheduler PASSED

kafka.utils.SchedulerTest > testPeriodicTask STARTED

kafka.utils.SchedulerTest > testPeriodicTask PASSED

kafka.utils.ZkUtilsTest > testAbortedConditionalDeletePath STARTED

kafka.utils.ZkUtilsTest > testAbortedConditionalDeletePath PASSED

kafka.utils.ZkUtilsTest > testSuccessfulConditionalDeletePath STARTED

kafka.utils.ZkUtilsTest > testSuccessfulConditionalDeletePath PASSED

kafka.utils.ZkUtilsTest > testPersistentSequentialPath STARTED

kafka.utils.ZkUtilsTest > testPersistentSequentialPath PASSED

kafka.utils.ZkUtilsTest > testClusterIdentifierJsonParsing STARTED

kafka.utils.ZkUtilsTest > testClusterIdentifierJsonParsing PASSED

kafka.utils.IteratorTemplateTest > testIterator STARTED

kafka.utils.IteratorTemplateTest > testIterator PASSED

kafka.utils.UtilsTest > testGenerateUuidAsBase64 STARTED

kafka.utils.UtilsTest > testGenerateUuidAsBase64 PASSED

kafka.utils.UtilsTest > testAbs STARTED

kafka.utils.UtilsTest > testAbs PASSED

kafka.utils.UtilsTest > testReplaceSuffix STARTED

kafka.utils.UtilsTest > testReplaceSuffix PASSED

kafka.utils.UtilsTest > testCircularIterator STARTED

kafka.utils.UtilsTest > testCircularIterator PASSED

kafka.utils.UtilsTest > testReadBytes STARTED

kafka.utils.UtilsTest > testReadBytes PASSED

kafka.utils.UtilsTest > testCsvList STARTED

kafka.utils.UtilsTest > testCsvList PASSED

kafka.utils.UtilsTest > testReadInt STARTED

kafka.utils.UtilsTest > testReadInt PASSED

kafka.utils.UtilsTest > testUrlSafeBase64EncodeUUID STARTED

kafka.utils.UtilsTest > testUrlSafeBase64EncodeUUID PASSED

kafka.utils.UtilsTest > testCsvMap STARTED

kafka.utils.UtilsTest > testCsvMap PASSED

kafka.utils.UtilsTest > testInLock STARTED

kafka.utils.UtilsTest > testInLock PASSED

kafka.utils.UtilsTest > testSwallow STARTED

kafka.utils.UtilsTest > testSwallow PASSED

kafka.producer.AsyncProducerTest > testFailedSendRetryLogic STARTED

kafka.producer.AsyncProducerTest > testFailedSendRetryLogic PASSED

kafka.producer.AsyncProducerTest > testQueueTimeExpired STARTED

kafka.producer.AsyncProducerTest > testQueueTimeExpired PASSED

kafka.producer.AsyncProducerTest > testPartitionAndCollateEvents STARTED

kafka.producer.AsyncProducerTest > testPartitionAndCollateEvents PASSED

kafka.producer.AsyncProducerTest > testBatchSize STARTED

kafka.producer.AsyncProducerTest > testBatchSize PASSED

kafka.producer.AsyncProducerTest > testSerializeEvents STARTED

kafka.producer.AsyncProducerTest > testSerializeEvents PASSED

kafka.producer.AsyncProducerTest > testProducerQueueSize STARTED

kafka.producer.AsyncProducerTest > testProducerQueueSize PASSED

kafka.producer.AsyncProducerTest > testRandomPartitioner STARTED

kafka.producer.AsyncProducerTest > testRandomPartitioner PASSED

kafka.producer.AsyncProducerTest > testInvalidConfiguration STARTED

kafka.producer.AsyncProducerTest > testInvalidConfiguration PASSED

kafka.producer.AsyncProducerTest > testInvalidPartition STARTED

kafka.producer.AsyncProducerTest > testInvalidPartition PASSED

kafka.producer.AsyncProducerTest > testNoBroker STARTED

kafka.producer.AsyncProducerTest > testNoBroker PASSED

kafka.producer.AsyncProducerTest > testProduceAfterClosed STARTED

kafka.producer.AsyncProducerTest > testProduceAfterClosed PASSED

kafka.producer.AsyncProducerTest > testJavaProducer STARTED

Re: Contributing to Kafka

Hi Rajan,

thanks for your plan to contribute back!

First, I would recommend to check out the wiki:
https://cwiki.apache.org/confluence/display/KAFKA/Contributing+Code+Changes

Also check out the JIRA boards:
https://issues.apache.org/jira/browse/KAFKA-1

You can just filter for components you are interested in and also filter
for JIRAs with label "newbie" (or similar) or priority "trival"/"minor"
to get started. If you found something interesting, just assign it to
yourself if unassigned (I guess you need to get permission first -- what
is you JIRA id? -- please share here at dev list, so we can add you as a
contributor). If a ticket is already assigned but it seems that nobody
is actively working on it, just post a comment and ask if you can talk
it over. All other question, can be discussed in the ticket comments, too.

Hope this helps to get started. Welcome to the community!

-Matthias

On 5/4/17 4:41 AM, Rajan Gupta wrote:
> Hello I have being leading teams using Kafka for processing high volume audit 
> and user behavior data and would like to give back to the Kafka community in 
> anyway that Dev likes. I can start by testing, write performance tests, 
> execute performance tests, monitor tests, fix bugs, write unit test code etc.
> Kindly advise how I can be of help. If there is a work stream where I can 
> pick up work from let me know and I will get started. I am self-starter so do 
> not require hand holding.
> Regards,RajanAtlanta, GA, USA
> 
> 

signature.asc
Description: OpenPGP digital signature

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

Thanks for updating the KIP.

Deep copying the key will work for sure, but I am actually a little bit
worried about performance impact... We might want to do some test to
quantify this impact.


Btw: this remind me about the idea of `RichFunction` interface that
would allow users to access record metadata (like timestamp, offset,
partition etc) within DSL. This would be a similar concept. Thus, I am
wondering, if it would make sense to enlarge the scope of this KIP by
that? WDYT?



-Matthias


On 5/3/17 2:08 AM, Jeyhun Karimov wrote:
> Hi Mathieu,
> 
> Thanks for feedback. I followed similar approach and updated PR and KIP
> accordingly. I tried to guard the key in Processors sending a copy of an
> actual key.
> Because I am doing deep copy of an object, I think memory can be bottleneck
> in some use-cases.
> 
> Cheers,
> Jeyhun
> 
> On Tue, May 2, 2017 at 5:10 PM Mathieu Fenniak 
> wrote:
> 
>> Hi Jeyhun,
>>
>> This approach would change ValueMapper (...etc) to be classes, rather than
>> interfaces, which is also a backwards incompatible change.  An alternative
>> approach that would be backwards compatible would be to define new
>> interfaces, and provide overrides where those interfaces are used.
>>
>> Unfortunately, making the key parameter as "final" doesn't change much
>> about guarding against key change.  It only prevents the parameter variable
>> from being reassigned.  If the key type is a mutable object (eg. byte[]),
>> it can still be mutated. (eg. key[0] = 0).  But I'm not really sure there's
>> much that can be done about that.
>>
>> Mathieu
>>
>>
>> On Mon, May 1, 2017 at 5:39 PM, Jeyhun Karimov 
>> wrote:
>>
>>> Thanks for comments.
>>>
>>> The concerns makes sense. Although we can guard for immutable keys in
>>> current implementation (with few changes), I didn't consider backward
>>> compatibility.
>>>
>>> In this case 2 solutions come to my mind. In both cases, user accesses
>> the
>>> key in Object type, as passing extra type parameter will break
>>> backwards-compatibility.  So user has to cast to actual key type.
>>>
>>> 1. Firstly, We can overload apply method with 2 argument (key and value)
>>> and force key to be *final*. By doing this,  I think we can address both
>>> backward-compatibility and guarding against key change.
>>>
>>> 2. Secondly, we can create class KeyAccess like:
>>>
>>> public class KeyAccess {
>>> Object key;
>>> public void beforeApply(final Object key) {
>>> this.key = key;
>>> }
>>> public Object getKey() {
>>> return key;
>>> }
>>> }
>>>
>>> We can extend *ValueMapper, ValueJoiner* and *ValueTransformer* from
>>> *KeyAccess*. Inside processor (for example *KTableMapValuesProcessor*)
>>> before calling *mapper.apply(value)* we can set the *key* by
>>> *mapper.beforeApply(key)*. As a result, user can use *getKey()* to access
>>> the key inside *apply(value)* method.
>>>
>>>
>>> Cheers,
>>> Jeyhun
>>>
>>>
>>>
>>>
>>> On Mon, May 1, 2017 at 7:24 PM Matthias J. Sax 
>>> wrote:
>>>
 Jeyhun,

 thanks a lot for the KIP!

 I think there are two issues we need to address:

 (1) The KIP does not consider backward compatibility. Users did
>> complain
 about this in past releases already, and as the user base grows, we
 should not break backward compatibility in future releases anymore.
 Thus, we should think of a better way to allow key access.

 Mathieu's comment goes into the same direction

>> On the other hand, the number of compile failures that would need to
>>> be
>> fixed from this change is unfortunate. :-)

 (2) Another concern is, that there is no guard to prevent user code to
 modify the key. This might corrupt partitioning if users do alter the
 key (accidentally -- or users are just not aware that they are not
 allowed to modify the provided key object) and thus break the
 application. (This was the original motivation to not provide the key
>> in
 the first place -- it's guards against modification.)


 -Matthias



 On 5/1/17 6:31 AM, Mathieu Fenniak wrote:
> Hi Jeyhun,
>
> I just want to add my voice that, I too, have wished for access to
>> the
> record key during a mapValues or similar operation.
>
> On the other hand, the number of compile failures that would need to
>> be
> fixed from this change is unfortunate. :-)  But at least it would all
>>> be
 a
> pretty clear and easy change.
>
> Mathieu
>
>
> On Mon, May 1, 2017 at 6:55 AM, Jeyhun Karimov >>
 wrote:
>
>> Dear community,
>>
>> I want to share KIP-149 [1] based on issues KAFKA-4218 [2],
>> KAFKA-4726
 [3],
>> KAFKA-3745 [4]. The related PR can be found at [5].
>> I would like to get your comments.
>>
>> [1]
>>

Re: [DISCUSS] KIP 141 - ProducerRecordBuilder Interface

We can go either way. I just pointed out, what I would prefer -- it's
also quite subjective.

The least invasive change would be to add new constructors and update
the JavaDocs to point out the semantics of `partition` parameter.

However, I still like the builder pattern: ProducerRecord has 6
parameters with only 2 being mandatory (topic and either key or value).
Thus, to have a complete set of overloads we would need many more
constructors. Right now, it feels to be "incomplete" and as if the
offered constructors got picked "randomly".

I got convinced though, that deprecation is not strictly required for
this change. If we go with option (2), it might be good to update the
JavaDocs of the current API to point to the new one as "recommended to use".



-Matthias


On 5/3/17 10:47 PM, Ewen Cheslack-Postava wrote:
> Stephane,
> 
> VOTES are really on-demand based on the author, but obviously it's good to
> come to some level of consensus in the DISCUSS thread before initiating a
> vote. I think the request for comments/votes on your 3 options is a
> reasonable way to gauge current opinions.
> 
> For myself, I think either 1 or 3 are good options, and I think at least
> Matthias & Jay are in agreement -- basically have one preferred, but
> possibly support 2 approaches for awhile.
> 
> I think 3 is the right way to go long term -- I don't expect so many more
> built-in fields to be added, but then again I didn't expect this much churn
> this quickly (headers were a surprise for me). We've gotten to enough
> parameters that a builder is more effective. It sucks a bit for existing
> users that rely on the constructors, but a full major release cycle (at the
> minimum) is a pretty significant window, and we can always choose to extend
> the window longer if we want to give people more time to transition. To me,
> the biggest problem is all the tutorials and content that we *can't*
> control -- there's a ton of code and tutorials out there that will still
> reference the constructors, and those will last far longer than any
> deprecation period we put in place.
> 
> -Ewen
> 
> On Wed, May 3, 2017 at 5:46 PM, Stephane Maarek <
> steph...@simplemachines.com.au> wrote:
> 
>> How do votes works?
>>
>> I feel there are 3 options right here, and I’d like a pre vote before a
>> real vote?
>> 1) Adding constructors. Could get messy over time, especially with headers
>> coming into play, and future possible improvement to the message format
>> 2) Adding a builder / nicer looking API (like fluent) to help build a
>> ProducerRecord in a safe way. Issue here are two ways of building a
>> ProducerRecord can bring confusion
>> 3) Same as 2), but deprecating all the constructors. May be too much of an
>> aggressive strategy
>>
>>
>> I’m happy to go over 2), update the docs, and tell people this is the
>> “preferred” way. Won’t outdate all the literature on Kafka, but I feel this
>> set people up for success in the future.
>> Thoughts  / pre vote?
>>
>> On 3/5/17, 4:20 pm, "Ewen Cheslack-Postava"  wrote:
>>
>> I understand the convenience of pointing at a JIRA/PR, but can we put
>> the
>> concrete changes proposed in the JIRA (under "Proposed Changes"). I
>> don't
>> think voting on the KIP would be reasonable otherwise since the changes
>> under vote could change arbitrarily...
>>
>> I'm increasingly skeptical of adding more convenience constructors --
>> the
>> current patch adds timestamps, we're about to add headers as well (for
>> core, for Connect we have
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> 145+-+Expose+Record+Headers+in+Kafka+Connect
>> in flight). It just continues to get messier over time.
>>
>> I think builders in the right context are useful, as long as they
>> exceed a
>> certain number of parameters (SchemaBuilder in Connect is an artifact
>> of
>> that position). I don't think a transition period with 2 ways to
>> construct
>> an object is actually a problem -- if there's always an "all N
>> parameters"
>> version of the constructor, all other constructors are just convenience
>> shortcuts, but the Builder provides a shorthand.
>>
>> I also agree w/ Ismael that deprecating to aggressively is bad -- we
>> added
>> the APIs instead of a builder and there's not any real maintenance
>> cost, so
>> why add the deprecation? I don't want to suggest actually adding such
>> an
>> annotation, but the real issue here is that one API will become
>> "preferred"
>> for some time.
>>
>> -Ewen
>>
>> On Tue, May 2, 2017 at 1:15 AM, Ismael Juma  wrote:
>>
>> > Hi Matthias,
>> >
>> > Deprecating widely used APIs is a big deal. Build warnings are a
>> nuisance
>> > and can potentially break the build for those who have a
>> zero-warnings
>> > policy (which is good practice). It creates a bunch of busy work for
>> our
>> > users and various resources like

Build failed in Jenkins: kafka-trunk-jdk7 #2148

See 


Changes:

[wangguoz] KAFKA-4925: delay initial rebalance of consumer group

--
[...truncated 844.64 KB...]
kafka.tools.ConsoleConsumerTest > shouldLimitReadsToMaxMessageLimit STARTED

kafka.tools.ConsoleConsumerTest > shouldLimitReadsToMaxMessageLimit PASSED

kafka.tools.ConsoleConsumerTest > shouldParseValidNewConsumerValidConfig STARTED

kafka.tools.ConsoleConsumerTest > shouldParseValidNewConsumerValidConfig PASSED

kafka.tools.ConsoleConsumerTest > shouldStopWhenOutputCheckErrorFails STARTED

kafka.tools.ConsoleConsumerTest > shouldStopWhenOutputCheckErrorFails PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewSimpleConsumerValidConfigWithStringOffset STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewSimpleConsumerValidConfigWithStringOffset PASSED

kafka.tools.ConsoleConsumerTest > shouldParseConfigsFromFile STARTED

kafka.tools.ConsoleConsumerTest > shouldParseConfigsFromFile PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewSimpleConsumerValidConfigWithNumericOffset STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewSimpleConsumerValidConfigWithNumericOffset PASSED

kafka.tools.ConsoleConsumerTest > testDefaultConsumer STARTED

kafka.tools.ConsoleConsumerTest > testDefaultConsumer PASSED

kafka.tools.ConsoleConsumerTest > shouldParseValidOldConsumerValidConfig STARTED

kafka.tools.ConsoleConsumerTest > shouldParseValidOldConsumerValidConfig PASSED

kafka.tools.ConsoleProducerTest > testParseKeyProp STARTED

kafka.tools.ConsoleProducerTest > testParseKeyProp PASSED

kafka.tools.ConsoleProducerTest > testValidConfigsOldProducer STARTED

kafka.tools.ConsoleProducerTest > testValidConfigsOldProducer PASSED

kafka.tools.ConsoleProducerTest > testInvalidConfigs STARTED

kafka.tools.ConsoleProducerTest > testInvalidConfigs PASSED

kafka.tools.ConsoleProducerTest > testValidConfigsNewProducer STARTED

kafka.tools.ConsoleProducerTest > testValidConfigsNewProducer PASSED

kafka.tools.MirrorMakerTest > 
testDefaultMirrorMakerMessageHandlerWithNoTimestampInSourceMessage STARTED

kafka.tools.MirrorMakerTest > 
testDefaultMirrorMakerMessageHandlerWithNoTimestampInSourceMessage PASSED

kafka.tools.MirrorMakerTest > testDefaultMirrorMakerMessageHandler STARTED

kafka.tools.MirrorMakerTest > testDefaultMirrorMakerMessageHandler PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytesWithCompression 
STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytesWithCompression 
PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > 
testIteratorIsConsistentWithCompression STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > 
testIteratorIsConsistentWithCompression PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testIteratorIsConsistent 
STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testIteratorIsConsistent PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testEqualsWithCompression 
STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testEqualsWithCompression 
PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testWrittenEqualsRead STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testWrittenEqualsRead PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testEquals STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testEquals PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytes STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytes PASSED

kafka.javaapi.consumer.ZookeeperConsumerConnectorTest > testBasic STARTED

kafka.javaapi.consumer.ZookeeperConsumerConnectorTest > testBasic PASSED

kafka.common.InterBrokerSendThreadTest > 
shouldCreateClientRequestAndSendWhenNodeIsReady STARTED

kafka.common.InterBrokerSendThreadTest > 
shouldCreateClientRequestAndSendWhenNodeIsReady PASSED

kafka.common.InterBrokerSendThreadTest > 
shouldCallCompletionHandlerWithDisconnectedResponseWhenNodeNotReady STARTED

kafka.common.InterBrokerSendThreadTest > 
shouldCallCompletionHandlerWithDisconnectedResponseWhenNodeNotReady PASSED

kafka.common.InterBrokerSendThreadTest > shouldNotSendAnythingWhenNoRequests 
STARTED

kafka.common.InterBrokerSendThreadTest > shouldNotSendAnythingWhenNoRequests 
PASSED

kafka.common.ConfigTest > testInvalidGroupIds STARTED

kafka.common.ConfigTest > testInvalidGroupIds PASSED

kafka.common.ConfigTest > testInvalidClientIds STARTED

kafka.common.ConfigTest > testInvalidClientIds PASSED

kafka.common.ZkNodeChangeNotificationListenerTest > testProcessNotification 
STARTED

kafka.common.ZkNodeChangeNotificationListenerTest > testProcessNotification 
PASSED

kafka.common.TopicTest > testInvalidTopicNames STARTED

kafka.common.TopicTest > testInvalidTopicNames PASSED

kafka.common.TopicTest > testTopicHasCollision STARTED

kafka.common.TopicTest > testTopicHasCollision PASSED

kafka.common.TopicTest >

[GitHub] kafka pull request #2976: KAFKA-5170. KafkaAdminClientIntegration test shoul...

2017-05-04 Thread cmccabe

GitHub user cmccabe opened a pull request:

https://github.com/apache/kafka/pull/2976

KAFKA-5170. KafkaAdminClientIntegration test should wait until metadaâ¦

â¦ta is propagated to all brokers

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cmccabe/kafka KAFKA-5170

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2976.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2976






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] KIP-138: Change punctuate semantics

Hi,

thanks for updating the KIP. Looks good to me overall.

I think adding `Cancellable` (or should it be `Cancelable` to follow
American English?) is a clean solution, in contrast to the proposed
alternative.

One minor comment: can you add `ValueTransformer#punctuate()` to the
list of deprecated methods?


-Matthias



On 5/4/17 1:41 AM, Michal Borowiecki wrote:
> Further in this direction I've updated the main proposal to incorporate
> the Cancellable return type for ProcessorContext.schedule and the
> guidance on how to implement "hybrid" punctuation with the proposed 2
> PunctuationTypes.
> 
> I look forward to more comments whether the Cancallable return type is
> an agreeable solution and it's precise definition.
> 
> I shall move all alternatives other than the main proposal into the
> Rejected Alternatives section and if I hear any objections, I'll move
> those back up and we'll discuss further.
> 
> 
> Looking forward to all comments and suggestions.
> 
> 
> Thanks,
> 
> Michal
> 
> 
> On 01/05/17 18:23, Michal Borowiecki wrote:
>>
>> Hi all,
>>
>> As promised, here is my take at how one could implement the previously
>> discussed hybrid semantics using the 2 PunctuationType callbacks (one
>> for STREAM_TIME and one for SYSTEM_TIME).
>>
>> However, there's a twist.
>>
>> Since currently calling context.schedule() adds a new
>> PunctuationSchedule and does not overwrite the previous one, a slight
>> change would be required:
>>
>> a) either that PuncuationSchedules are cancellable
>>
>> b) or that calling schedule() ||overwrites(cancels) the previous one
>> with the given |PunctuationType |(but that's not how it works currently)
>>
>>
>> Below is an example assuming approach a) is implemented by having
>> schedule return Cancellable instead of void.
>>
>> |ProcessorContext context;|
>> |long| |streamTimeInterval = ...;|
>> |long| |systemTimeUpperBound = ...; ||//e.g. systemTimeUpperBound =
>> streamTimeInterval + some tolerance|
>> |Cancellable streamTimeSchedule;|
>> |Cancellable systemTimeSchedule;|
>> |long| |lastStreamTimePunctation = -||1||;|
>> | |
>> |public| |void| |init(ProcessorContext context){|
>> |||this||.context = context;|
>> |||streamTimeSchedule =
>> context.schedule(PunctuationType.STREAM_TIME,
>> streamTimeInterval,   ||this||::streamTimePunctuate);|
>> |||systemTimeSchedule =
>> context.schedule(PunctuationType.SYSTEM_TIME,
>> systemTimeUpperBound, ||this||::systemTimePunctuate);   |
>> |}|
>> | |
>> |public| |void| |streamTimePunctuate(||long| |streamTime){|
>> |||periodicBusiness(streamTime);|
>>  
>> |||systemTimeSchedule.cancel();|
>> |||systemTimeSchedule =
>> context.schedule(PunctuationType.SYSTEM_TIME,
>> systemTimeUpperBound, ||this||::systemTimePunctuate);|
>> |}|
>> | |
>> |public| |void| |systemTimePunctuate(||long| |systemTime){|
>> |||periodicBusiness(context.timestamp());|
>>  
>> |||streamTimeSchedule.cancel();|
>> |||streamTimeSchedule =
>> context.schedule(PunctuationType.STREAM_TIME,
>> streamTimeInterval, ||this||::streamTimePunctuate);|
>> |}|
>> | |
>> |public| |void| |periodicBusiness(||long| |streamTime){|
>> |||// guard against streamTime == -1, easy enough.|
>> |||// if you need system time instead, just use
>> System.currentTimeMillis()|
>> | |
>> |||// do something businessy here|
>> |}|
>>
>> Where Cancellable is either an interface containing just a single void
>> cancel() method or also boolean isCancelled() like here
>> .
>>
>>
>> Please let your opinions known whether we should proceed in this
>> direction or leave "hybrid" considerations out of scope.
>>
>> Looking forward to hearing your thoughts.
>>
>> Thanks,
>> Michal
>>
>> On 30/04/17 20:07, Michal Borowiecki wrote:
>>>
>>> Hi Matthias,
>>>
>>> I'd like to start moving the discarded ideas into Rejected
>>> Alternatives section. Before I do, I want to tidy them up, ensure
>>> they've each been given proper treatment.
>>>
>>> To that end let me go back to one of your earlier comments about the
>>> original suggestion (A) to put that to bed.
>>>
>>>
>>> On 04/04/17 06:44, Matthias J. Sax wrote:
 (A) You argue, that users can still "punctuate" on event-time via
 process(), but I am not sure if this is possible. Note, that users only
 get record timestamps via context.timestamp(). Thus, users would need to
 track the time progress per partition (based on the partitions they
 obverse via context.partition(). (This alone puts a huge burden on the
 user by itself.) However, users are not notified at startup what
 partitions are assigned, and user are not notified when partitions get
 revoked. Because this information is not available, it's not possible to
 "manually advance" stream-time, and thus event-time punctuation within
 process() seems not to be possible -- or do you see a way to get it
 done? And even if, it might still be too clumsy

[jira] [Commented] (KAFKA-5170) KafkaAdminClientIntegration test should wait until metadata is propagated to all brokers


[ 
https://issues.apache.org/jira/browse/KAFKA-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997329#comment-15997329
 ] 

ASF GitHub Bot commented on KAFKA-5170:
---

GitHub user cmccabe opened a pull request:

https://github.com/apache/kafka/pull/2976

KAFKA-5170. KafkaAdminClientIntegration test should wait until metada…

…ta is propagated to all brokers

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cmccabe/kafka KAFKA-5170

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2976.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2976






> KafkaAdminClientIntegration test should wait until metadata is propagated to 
> all brokers
> 
>
> Key: KAFKA-5170
> URL: https://issues.apache.org/jira/browse/KAFKA-5170
> Project: Kafka
>  Issue Type: Bug
>  Components: admin
>Affects Versions: 0.11.0.0
>Reporter: Colin P. McCabe
>Assignee: Colin P. McCabe
> Fix For: 0.11.0.0
>
>
> The KafkaAdminClientIntegration test and its subclasses should wait until the 
> metadata is propagated to all brokers.  We have seen a few test failures that 
> resulted from some brokers having partial metadata.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2905: kafka-5104: DumpLogSegments should not open index ...

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2905


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (KAFKA-5104) DumpLogSegments should not open index files with `rw`


 [ 
https://issues.apache.org/jira/browse/KAFKA-5104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-5104.
--
   Resolution: Fixed
Fix Version/s: 0.11.0.0

Issue resolved by pull request 2905
[https://github.com/apache/kafka/pull/2905]

> DumpLogSegments should not open index files with `rw`
> -
>
> Key: KAFKA-5104
> URL: https://issues.apache.org/jira/browse/KAFKA-5104
> Project: Kafka
>  Issue Type: Improvement
>  Components: log
>Affects Versions: 0.10.2.0
> Environment: Kafka is run as root
>Reporter: Yeva Byzek
>Assignee: huxi
>Priority: Minor
> Fix For: 0.11.0.0
>
>
> kafka.tools.DumpLogSegments requires sudo for index files but not for log 
> files. It seems inconsistent why `w` access would be required for the index 
> files just to dump the output.
> {noformat}
> $ sudo kafka-run-class kafka.tools.DumpLogSegments  --files 
> .index
> Dumping .index
> offset: 0 position: 0
> {noformat}
> {noformat}
> $ kafka-run-class kafka.tools.DumpLogSegments  --files 
> .indexDumping .index
> Exception in thread "main" java.io.FileNotFoundException: 
> .index (Permission denied)
> at java.io.RandomAccessFile.open0(Native Method)
> at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
> at java.io.RandomAccessFile.(RandomAccessFile.java:243)
> at kafka.log.AbstractIndex.(AbstractIndex.scala:50)
> at kafka.log.OffsetIndex.(OffsetIndex.scala:52)
> at 
> kafka.tools.DumpLogSegments$.kafka$tools$DumpLogSegments$$dumpIndex(DumpLogSegments.scala:137)
> at 
> kafka.tools.DumpLogSegments$$anonfun$main$1.apply(DumpLogSegments.scala:100)
> at 
> kafka.tools.DumpLogSegments$$anonfun$main$1.apply(DumpLogSegments.scala:93)
> at 
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
> at kafka.tools.DumpLogSegments$.main(DumpLogSegments.scala:93)
> at kafka.tools.DumpLogSegments.main(DumpLogSegments.scala)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: kafka-trunk-jdk8 #1481

See 


Changes:

[wangguoz] KAFKA-4925: delay initial rebalance of consumer group

--
[...truncated 844.34 KB...]

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokerListWithNoTopics STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokerListWithNoTopics PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testAutoCreateTopic STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testGetAllTopicMetadata 
STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testGetAllTopicMetadata 
PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testBasicTopicMetadata 
STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.integration.PrimitiveApiTest > testMultiProduce STARTED

kafka.integration.PrimitiveApiTest > testMultiProduce PASSED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch STARTED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch PASSED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize 
STARTED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize PASSED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests STARTED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests PASSED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch STARTED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch PASSED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression STARTED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression PASSED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic STARTED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic PASSED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest STARTED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack PASSED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopicWithCollision 
STARTED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopicWithCollision 
PASSED

kafka.integration.PlaintextTopicMetadataTest > testAliveBrokerListWithNoTopics 
STARTED

kafka.integration.PlaintextTopicMetadataTest > testAliveBrokerListWithNoTopics 
PASSED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopic STARTED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.PlaintextTopicMetadataTest > testGetAllTopicMetadata STARTED

kafka.integration.PlaintextTopicMetadataTest > testGetAllTopicMetadata PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.PlaintextTopicMetadataTest > testBasicTopicMetadata STARTED

kafka.integration.PlaintextTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.integration.MetricsDuringTopicCreationDeletionTest > 
testMetricsDuringTopicCreateDelete STARTED

kafka.integration.MetricsDuringTopicCreationDeletionTest > 
testMetricsDuringTopicCreateDelete PASSED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooLow 
STARTED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooLow PASSED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooLow 
STARTED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooLow 
PASSED

[jira] [Assigned] (KAFKA-5174) RocksDb might stall in environments with 1 core only

2017-05-04 Thread Eno Thereska (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-5174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eno Thereska reassigned KAFKA-5174:
---

Assignee: Eno Thereska

> RocksDb might stall in environments with 1 core only
> 
>
> Key: KAFKA-5174
> URL: https://issues.apache.org/jira/browse/KAFKA-5174
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.1
>Reporter: Eno Thereska
>Assignee: Eno Thereska
>
> - there is a regression in 0.10.2.1 that impacts RocksDb's behaviour (details 
> below for first impact) from this PR: 
> https://github.com/apache/kafka/commit/f9660e16dafb9e1109704bdecb978b7046f0cc6b#diff-046ca566243518c88e007b7499ec9f51
> - worst-case impact: streams might not run out-of-box in deployments with 1 
> core only
> - workaround: no new jars needed for the workaround, only a config change 
> that changes a RocksDb parameter, e.g. something like in this PR: 
> https://github.com/confluentinc/examples/pull/117/files
> - full fix: add 1 to the current config (i.e., max cpus + 1) in the default 
> settings
> The RocksDb code might have a bug itself, since when running with one core
> it both sets the number of compaction threads to 1 
> (https://github.com/facebook/rocksdb/blob/5.0.fb/util/options.cc#L762) AND 
> the maximum number of compactions variable to 0 
> (https://github.com/facebook/rocksdb/blob/5.0.fb/util/options.cc#L760) which 
> doesn't make much sense. I've sent a message in RocksDb channel.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (KAFKA-5174) RocksDb might stall in environments with 1 core only

2017-05-04 Thread Eno Thereska (JIRA)

Eno Thereska created KAFKA-5174:
---

 Summary: RocksDb might stall in environments with 1 core only
 Key: KAFKA-5174
 URL: https://issues.apache.org/jira/browse/KAFKA-5174
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 0.10.2.1
Reporter: Eno Thereska


- there is a regression in 0.10.2.1 that impacts RocksDb's behaviour (details 
below for first impact) from this PR: 
https://github.com/apache/kafka/commit/f9660e16dafb9e1109704bdecb978b7046f0cc6b#diff-046ca566243518c88e007b7499ec9f51

- worst-case impact: streams might not run out-of-box in deployments with 1 
core only
- workaround: no new jars needed for the workaround, only a config change that 
changes a RocksDb parameter, e.g. something like in this PR: 
https://github.com/confluentinc/examples/pull/117/files


- full fix: add 1 to the current config (i.e., max cpus + 1) in the default 
settings

The RocksDb code might have a bug itself, since when running with one core
it both sets the number of compaction threads to 1 
(https://github.com/facebook/rocksdb/blob/5.0.fb/util/options.cc#L762) AND the 
maximum number of compactions variable to 0 
(https://github.com/facebook/rocksdb/blob/5.0.fb/util/options.cc#L760) which 
doesn't make much sense. I've sent a message in RocksDb channel.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (KAFKA-5118) Improve message for Kafka failed startup with non-Kafka data in data.dirs


 [ 
https://issues.apache.org/jira/browse/KAFKA-5118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-5118.
--
   Resolution: Fixed
Fix Version/s: 0.11.0.0

Issue resolved by pull request 2907
[https://github.com/apache/kafka/pull/2907]

> Improve message for Kafka failed startup with non-Kafka data in data.dirs
> -
>
> Key: KAFKA-5118
> URL: https://issues.apache.org/jira/browse/KAFKA-5118
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.10.2.0
>Reporter: Dustin Cote
>Assignee: huxi
>Priority: Minor
> Fix For: 0.11.0.0
>
>
> Today, if you try to startup a broker with some non-Kafka data in the 
> data.dirs you end up with a cryptic message:
> {code}
> [2017-04-21 13:35:08,122] ERROR There was an error in one of the threads 
> during logs loading: java.lang.StringIndexOutOfBoundsException: String index 
> out of range: -1 (kafka.log.LogManager) 
> [2017-04-21 13:35:08,124] FATAL [Kafka Server 3], Fatal error during 
> KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) 
> java.lang.StringIndexOutOfBoundsException: String index out of range: -1 
> {code}
> It'd be better if we could tell the user to look for non-Kafka data in the 
> data.dirs and print out the offending directory that caused the problem in 
> the first place.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2907: kafka-5118: Improve message for Kafka failed start...

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2907


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

2017-05-04 Thread Kyle Winkelman

I have made a pull request. It can be found here.

https://github.com/apache/kafka/pull/2975

I plan to write some more unit tests for my classes and get around to
writing documentation for the public api additions.

One thing I was curious about is during the KCogroupedStreamImpl#aggregate
method I pass null to the KGroupedStream#repartitionIfRequired method. I
can't supply the store name because if more than one grouped stream
repartitions an error is thrown. Is there some name that someone can
recommend or should I leave the null and allow it to fall back to the
KGroupedStream.name?

Should this be expanded to handle grouped tables? This would be pretty easy
for a normal aggregate but one allowing session stores and windowed stores
would required KTableSessionWindowAggregate and KTableWindowAggregate
implementations.

Thanks,
Kyle

On May 4, 2017 1:24 PM, "Eno Thereska"  wrote:

> I’ll look as well asap, sorry, been swamped.
>
> Eno
> > On May 4, 2017, at 6:17 PM, Damian Guy  wrote:
> >
> > Hi Kyle,
> >
> > Thanks for the KIP. I apologize that i haven't had the chance to look at
> > the KIP yet, but will schedule some time to look into it tomorrow. For
> the
> > implementation, can you raise a PR against kafka trunk and mark it as
> WIP?
> > It will be easier to review what you have done.
> >
> > Thanks,
> > Damian
> >
> > On Thu, 4 May 2017 at 11:50 Kyle Winkelman 
> wrote:
> >
> >> I am replying to this in hopes it will draw some attention to my KIP as
> I
> >> haven't heard from anyone in a couple days. This is my first KIP and my
> >> first large contribution to the project so I'm sure I did something
> wrong.
> >> ;)
> >>
> >> On May 1, 2017 4:18 PM, "Kyle Winkelman" 
> wrote:
> >>
> >>> Hello all,
> >>>
> >>> I have created KIP-150 to facilitate discussion about adding cogroup to
> >>> the streams DSL.
> >>>
> >>> Please find the KIP here:
> >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>> 150+-+Kafka-Streams+Cogroup
> >>>
> >>> Please find my initial implementation here:
> >>> https://github.com/KyleWinkelman/kafka
> >>>
> >>> Thanks,
> >>> Kyle Winkelman
> >>>
> >>
>
>

Re: [DISCUSS] KIP-147: Add missing type parameters to StateStoreSupplier factories and KGroupedStream/Table methods

Did not have time to have a look. But backward compatibility is a must
from my point of view.

-Matthias


On 5/4/17 12:56 AM, Michal Borowiecki wrote:
> Hello,
> 
> I've updated the KIP with missing information.
> 
> I would especially appreciate some comments on the compatibility aspects
> of this as the proposed change is not fully backwards-compatible.
> 
> In the absence of comments I shall call for a vote in the next few days.
> 
> Thanks,
> 
> Michal
> 
> 
> On 30/04/17 23:11, Michal Borowiecki wrote:
>>
>> Hi community!
>>
>> I have just drafted KIP-147: Add missing type parameters to
>> StateStoreSupplier factories and KGroupedStream/Table methods
>> 
>>
>> Please let me know if this a step in the right direction.
>>
>> All comments welcome.
>>
>> Thanks,
>> Michal
>> -- 
>> Signature
>> Michal Borowiecki
>> Senior Software Engineer L4
>>  T:  +44 208 742 1600
>>
>>  
>>  +44 203 249 8448
>>
>>  
>>   
>>  E:  michal.borowie...@openbet.com
>>  W:  www.openbet.com 
>>
>>  
>>  OpenBet Ltd
>>
>>  Chiswick Park Building 9
>>
>>  566 Chiswick High Rd
>>
>>  London
>>
>>  W4 5XT
>>
>>  UK
>>
>>  
>> 
>>
>> This message is confidential and intended only for the addressee. If
>> you have received this message in error, please immediately notify the
>> postmas...@openbet.com  and delete it
>> from your system as well as any copies. The content of e-mails as well
>> as traffic data may be monitored by OpenBet for employment and
>> security purposes. To protect the environment please do not print this
>> e-mail unless necessary. OpenBet Ltd. Registered Office: Chiswick Park
>> Building 9, 566 Chiswick High Road, London, W4 5XT, United Kingdom. A
>> company registered in England and Wales. Registered no. 3134634. VAT
>> no. GB927523612
>>
> 
> -- 
> Signature
>  Michal Borowiecki
> Senior Software Engineer L4
>   T:  +44 208 742 1600
> 
>   
>   +44 203 249 8448
> 
>   
>
>   E:  michal.borowie...@openbet.com
>   W:  www.openbet.com 
> 
>   
>   OpenBet Ltd
> 
>   Chiswick Park Building 9
> 
>   566 Chiswick High Rd
> 
>   London
> 
>   W4 5XT
> 
>   UK
> 
>   
> 
> 
> This message is confidential and intended only for the addressee. If you
> have received this message in error, please immediately notify the
> postmas...@openbet.com  and delete it
> from your system as well as any copies. The content of e-mails as well
> as traffic data may be monitored by OpenBet for employment and security
> purposes. To protect the environment please do not print this e-mail
> unless necessary. OpenBet Ltd. Registered Office: Chiswick Park Building
> 9, 566 Chiswick High Road, London, W4 5XT, United Kingdom. A company
> registered in England and Wales. Registered no. 3134634. VAT no.
> GB927523612
> 



signature.asc
Description: OpenPGP digital signature

[jira] [Commented] (KAFKA-4925) Add a configurable delay to the initial consumer group rebalance


[ 
https://issues.apache.org/jira/browse/KAFKA-4925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997160#comment-15997160
 ] 

ASF GitHub Bot commented on KAFKA-4925:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2758


> Add a configurable delay to the initial consumer group rebalance
> 
>
> Key: KAFKA-4925
> URL: https://issues.apache.org/jira/browse/KAFKA-4925
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.2.0
>Reporter: Damian Guy
>Assignee: Damian Guy
> Fix For: 0.11.0.0
>
>
> As per 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-134%3A+Delay+initial+consumer+group+rebalance
> Add a broker side config that will enable the GroupCoordinator to delay the 
> initial rebalance for a new/empty consumer group



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2758: KAFKA-4925: delay initial rebalance of consumer gr...

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2758


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-4925) Add a configurable delay to the initial consumer group rebalance