Different key with the same digest in log compaction

[Renkai]

Hi, all:

 I am just learning the Kafka codebase, as what I saw in 
https://github.com/apache/kafka/blob/6ed3e6b1cb8a73b1f5f78926ccb247a8953a554c/core/src/main/scala/kafka/log/OffsetMap.scala#L43-L43
 

if different log keys have the same digest value, they will be treated as the 
same key in log compaction. Though the risk of such things happens is very 
small, I still want it to be avoided. If what I thought is wrong please let me 
know, and I hope to know the thoughts of who created or is maintaining the code.

 

Different key with the same digest in log compaction

[Renkai Ge]

Hi,all:
 I am just learning the kafka codebase, as what I saw in
https://github.com/apache/kafka/blob/6ed3e6b1cb8a73b1f5f78926ccb247a8953a554c/core/src/main/scala/kafka/log/OffsetMap.scala#L43-L43

if different log keys have the same digest value, they will be treated as
the same key in log compaction.Though the risk of such things happens is
very small, I still want it to be avoided.If what I thought is wrong please
let me know, and I hope to know the thoughts of who created or
is maintaining the code.

[jira] [Commented] (KAFKA-4560) Min / Max Partitions Fetch Records params

[huxi (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15769203#comment-15769203
 ] 

huxi commented on KAFKA-4560:
-

Could you illustrate a real scenario when 'max.partitions.fetch.records' might 
be a help whereas 'max.poll.records' cannot be used?

> Min / Max Partitions Fetch Records params
> -
>
> Key: KAFKA-4560
> URL: https://issues.apache.org/jira/browse/KAFKA-4560
> Project: Kafka
>  Issue Type: Improvement
>  Components: consumer
>Affects Versions: 0.10.0.1
>Reporter: Stephane Maarek
>  Labels: features, newbie
>
> There is currently a `max.partition.fetch.bytes` parameter to limit the total 
> size of the fetch call (also a min).
> Sometimes I'd like to control how many records altogether I'm getting at the 
> time and I'd like to see a `max.partition.fetch.records` (also a min).
> If both are specified the first condition that is met would complete the 
> fetch call. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-4566) Can't Symlink to Kafka bins

[Stephane Maarek (JIRA)]

Stephane Maarek created KAFKA-4566:
--

 Summary: Can't Symlink to Kafka bins
 Key: KAFKA-4566
 URL: https://issues.apache.org/jira/browse/KAFKA-4566
 Project: Kafka
  Issue Type: Bug
  Components: tools
Affects Versions: 0.10.1.1
Reporter: Stephane Maarek


in the kafka consumer for example, the last line is :
https://github.com/apache/kafka/blob/trunk/bin/kafka-console-consumer.sh#L21

{code}
exec $(dirname $0)/kafka-run-class.sh kafka.tools.ConsoleConsumer "$@"
{code}

if I create a symlink using 
{code}
ln -s
{code}

it doesn't resolve the right directory name because of $(dirname $0) 

I believe the right way is to do:
{code}
"$(dirname "$(readlink -e "$0")")"
{code}
 

Any thoughts on that before I do a PR?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[ANNOUCE] Apache Kafka 0.10.1.1 Released

[Guozhang Wang]

The Apache Kafka community is pleased to announce the release for Apache
Kafka 0.10.1.1. This is a bug fix release that fixes 30 issues in 0.10.1.0.

All of the changes in this release can be found in the release notes:
*https://archive.apache.org/dist/kafka/0.10.1.1/RELEASE_NOTES.html


Apache Kafka is a distributed streaming platform with four four core APIs:

** The Producer API allows an application to publish a stream records to
one or more Kafka topics.

** The Consumer API allows an application to subscribe to one or more
topics and process the stream of records produced to them.

** The Streams API allows an application to act as a stream processor,
consuming an input stream from one or more topics and producing an output
stream to one or more output topics, effectively transforming the input
streams to output streams.

** The Connector API allows building and running reusable producers or
consumers that connect Kafka topics to existing applications or data
systems. For example, a connector to a relational database might capture
every change to a table.three key capabilities:


With these APIs, Kafka can be used for two broad classes of application:

** Building real-time streaming data pipelines that reliably get data
between systems or applications.

** Building real-time streaming applications that transform or react to the
streams of data.


You can download the source release from
https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.1.1/kafka-0.10.1.1-src.tgz

and binary releases from
https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.1.1/kafka_2.10-0.10.1.1.tgz
https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.1.1/kafka_2.11-0.10.1.1.tgz


A big thank you for the following 21 contributors to this release!

Alexey Ozeritsky, Anton Karamanov, Ben Stopford, Bernard Leach, Bill
Bejeck, Damian Guy, Dan Norwood, Eno Thereska, Ewen Cheslack-Postava,
Guozhang Wang, Jason Gustafson, Jiangjie Qin, Jun He, Jun Rao, Kim
Christensen, Manikumar Reddy O, Matthias J. Sax, Mayuresh Gharat, Rajini
Sivaram, Sumant Tambe, Vahid Hashemian

We welcome your help and feedback. For more information on how to
report problems, and to get involved, visit the project website at
http://kafka.apache.org/


Thanks,
-- Guozhang

Jenkins build is back to normal : kafka-trunk-jdk8 #1126

[Apache Jenkins Server]

See 

[GitHub] kafka pull request #2288: MINOR: Create Hyperlinks in protocol api docs

[imandhan]

GitHub user imandhan opened a pull request:

https://github.com/apache/kafka/pull/2288

MINOR: Create Hyperlinks in protocol api docs

Minor changes to generate hyperlinks in the protocol api documentation to 
ease lookup.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/imandhan/kafka HyperLink

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2288.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2288


commit d2afdc4d917ab4abeba4cbde5e4a9dd8eb985f7a
Author: Ishita Mandhan 
Date:   2016-12-22T03:18:53Z

MINOR: Create Hyperlinks in protocol api docs




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4277) creating ephemeral node already exist

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15768759#comment-15768759
 ] 

Jun Rao commented on KAFKA-4277:


[~fpj], in KAFKA-1387, you had the following comment:
"If a client has received a session expiration event, it means that the leader 
has expired the session and has broadcast the closeSession event to the 
followers. If the same client creates a new session successfully, then the 
server it connects to must have applied the previous closeSession, which 
deletes the ephemeral znodes, because ZK guarantees that txns are totally 
ordered. Consequently, the client shouldn't observe an ephemeral from an old 
session of its own. Note that another client could still observe the ephemeral 
znode after the session expiration if it is connected to a server that is a bit 
behind, but that's fine."

However, a while back in ZK's mailing list, you also had the following comment 
(http://zookeeper.markmail.org/search/?q=Zookeeper+3.3.4#query:Zookeeper%203.3.4%20date%3A201307%20+page:1+mid:zma242a2qgp6gxvx+state:results).

"Unless we expire a session and delete ephemerals atomically, there are only
two options I see:

1- Delete right before expiring the session
2- Delete right after expiring the session

Because of timing, we can have the following. With the first, a client might
observe the delete before the session actually expires, which violates our
contract. With the second, you may observe an ephemeral znode after the
session has expired as you have. I would say that the second option is
correct as long as the ephemerals are eventually deleted, but it does have
the side-effect you're mentioning."

Could you clarify whether ZK server guarantees to delete ephemeral nodes before 
notifying a client about session expiration?

> creating ephemeral node already exist
> -
>
> Key: KAFKA-4277
> URL: https://issues.apache.org/jira/browse/KAFKA-4277
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.0.0
>Reporter: Feixiang Yan
>
> I use zookeeper 3.4.6.
> Zookeeper session time out, zkClient try reconnect failed. Then re-establish 
> the session and re-registering broker info in ZK, throws NODEEXISTS Exception.
>  I think it is because the ephemeral node which created by old session has 
> not removed. 
> I read the 
> [ZkUtils.scala|https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/utils/ZkUtils.scala]
>  of 0.8.1, createEphemeralPathExpectConflictHandleZKBug try create node in a 
> while loop until create success. This can solve the issue. But in 
> [ZkUtils.scala|https://github.com/apache/kafka/blob/0.10.0.1/core/src/main/scala/kafka/utils/ZkUtils.scala]
>   0.10.1 the function removed.
> {noformat}
> [2016-10-07 19:00:32,562] INFO Socket connection established to 
> 10.191.155.238/10.191.155.238:21819, initiating session 
> (org.apache.zookeeper.ClientCnxn)
> [2016-10-07 19:00:32,563] INFO zookeeper state changed (Expired) 
> (org.I0Itec.zkclient.ZkClient)
> [2016-10-07 19:00:32,564] INFO Unable to reconnect to ZooKeeper service, 
> session 0x1576b11f9b201bd has expired, closing socket connection 
> (org.apache.zookeeper.ClientCnxn)
> [2016-10-07 19:00:32,564] INFO Initiating client connection, 
> connectString=10.191.155.237:21819,10.191.155.238:21819,10.191.155.239:21819/cluster2
>  sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@ae71be2 
> (org.apache.zookeeper.ZooKeeper)
> [2016-10-07 19:00:32,566] INFO Opening socket connection to server 
> 10.191.155.237/10.191.155.237:21819. Will not attempt to authenticate using 
> SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
> [2016-10-07 19:00:32,566] INFO Socket connection established to 
> 10.191.155.237/10.191.155.237:21819, initiating session 
> (org.apache.zookeeper.ClientCnxn)
> [2016-10-07 19:00:32,566] INFO EventThread shut down 
> (org.apache.zookeeper.ClientCnxn)
> [2016-10-07 19:00:32,567] INFO Session establishment complete on server 
> 10.191.155.237/10.191.155.237:21819, sessionid = 0x1579ecd39c20006, 
> negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
> [2016-10-07 19:00:32,567] INFO zookeeper state changed (SyncConnected) 
> (org.I0Itec.zkclient.ZkClient)
> [2016-10-07 19:00:32,608] INFO re-registering broker info in ZK for broker 3 
> (kafka.server.KafkaHealthcheck$SessionExpireListener)
> [2016-10-07 19:00:32,610] INFO Creating /brokers/ids/3 (is it secure? false) 
> (kafka.utils.ZKCheckedEphemeral)
> [2016-10-07 19:00:32,611] INFO Result of znode creation is: NODEEXISTS 
> (kafka.utils.ZKCheckedEphemeral)
> [2016-10-07 19:00:32,614] ERROR Error handling event ZkEvent[New session 
> event sent to kafka.server.KafkaHealthcheck$SessionExpireListener@324f1bc] 
> (org.I0Itec.zkclient.ZkEventThread)
> java.lang.RuntimeException: A broker is already registered on the p

[jira] [Commented] (KAFKA-2273) Add rebalance with a minimal number of reassignments to server-defined strategy list

[Vahid Hashemian (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15768722#comment-15768722
 ] 

Vahid Hashemian commented on KAFKA-2273:


The thread is 
[here|https://www.mail-archive.com/dev@kafka.apache.org/msg56583.html].
It has one binding vote, and requires two more to proceed.

> Add rebalance with a minimal number of reassignments to server-defined 
> strategy list
> 
>
> Key: KAFKA-2273
> URL: https://issues.apache.org/jira/browse/KAFKA-2273
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Olof Johansson
>Assignee: Vahid Hashemian
>  Labels: newbie++, newbiee
> Fix For: 0.10.2.0
>
>
> Add a new partitions.assignment.strategy to the server-defined list that will 
> do reassignments based on moving as few partitions as possible. This should 
> be a quite common reassignment strategy especially for the cases where the 
> consumer has to maintain state, either in memory, or on disk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2273) Add rebalance with a minimal number of reassignments to server-defined strategy list

[Jeff Widman (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15768710#comment-15768710
 ] 

Jeff Widman commented on KAFKA-2273:


What was the outcome of the vote?

> Add rebalance with a minimal number of reassignments to server-defined 
> strategy list
> 
>
> Key: KAFKA-2273
> URL: https://issues.apache.org/jira/browse/KAFKA-2273
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Olof Johansson
>Assignee: Vahid Hashemian
>  Labels: newbie++, newbiee
> Fix For: 0.10.2.0
>
>
> Add a new partitions.assignment.strategy to the server-defined list that will 
> do reassignments based on moving as few partitions as possible. This should 
> be a quite common reassignment strategy especially for the cases where the 
> consumer has to maintain state, either in memory, or on disk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk8 #1125

[Apache Jenkins Server]

See 

Changes:

[wangguoz] MINOR: trivial logCleanerEnable doc cleanup

[wangguoz] MINOR: Fix document header/footer links

--
[...truncated 30479 lines...]
kafka.integration.SaslPlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithCollision STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithCollision PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokerListWithNoTopics STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokerListWithNoTopics PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testAutoCreateTopic STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testGetAllTopicMetadata 
STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testGetAllTopicMetadata 
PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testBasicTopicMetadata 
STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.integration.PrimitiveApiTest > testMultiProduce STARTED

kafka.integration.PrimitiveApiTest > testMultiProduce PASSED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch STARTED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch PASSED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize 
STARTED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize PASSED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests STARTED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests PASSED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch STARTED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch PASSED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression STARTED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression PASSED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic STARTED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic PASSED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest STARTED
ERROR: Could not install GRADLE_2_4_RC_2_HOME
java.lang.NullPointerException

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack PASSED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopicWithCollision 
STARTED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopicWithCollision 
PASSED

kafka.integration.PlaintextTopicMetadataTest > testAliveBrokerListWithNoTopics 
STARTED

kafka.integration.PlaintextTopicMetadataTest > testAliveBrokerListWithNoTopics 
PASSED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopic STARTED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.PlaintextTopicMetadataTest > testGetAllTopicMetadata STARTED

kafka.integration.PlaintextTopicMetadataTest > testGetAllTopicMetadata PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.PlaintextTopicMetadataTest > testBasicTopicMetadata STARTED

kafka.integration.PlaintextTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooLow 
START

Re: [VOTE] KIP-92 - Add per partition lag metrics to KafkaConsumer

[Becket Qin]

Just updated the wiki to clarify that.

Thanks everyone for the votes. KIP-92 has passed with +3 (binding) and +2
(non-binding).

On Wed, Dec 21, 2016 at 1:53 PM, Ismael Juma  wrote:

> That seems reasonable. It would be good to update the KIP to make this
> clear.
>
> Ismael
>
> On Wed, Dec 21, 2016 at 1:49 PM, Becket Qin  wrote:
>
> > Hi Ismael,
> >
> > Thanks for the comments. Good observation. I guess for max lag of all the
> > partitions the average value is less meaningful because the lag can be
> from
> > different partitions, so an average of lags from different partitions at
> > different times seems hard to reason about. On the other hand, for per
> > partition lag, the average value of the same partition at different times
> > seems having some real meaning?
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> > On Wed, Dec 21, 2016 at 1:02 PM, Ismael Juma  wrote:
> >
> > > Thanks for the KIP, it's a useful improvement. Just one question, the
> KIP
> > > states that it's about adding per partition metrics for metrics that
> > > already exist. However, when I look at the code, it seems that
> > > `records-lag-avg` doesn't exist?
> > >
> > > this.recordsFetchLag = metrics.sensor("records-lag");
> > > this.recordsFetchLag.add(metrics.metricName("records-lag-max",
> > > this.metricGrpName, "The maximum lag in terms of number of records for
> > any
> > > partition in this window"), new Max());
> > >
> > > Ismael
> > >
> > > On Thu, Dec 15, 2016 at 5:32 PM, Becket Qin 
> > wrote:
> > >
> > > > Hi,
> > > >
> > > > I want to start a voting thread on KIP-92 which proposes to add per
> > > > partition lag metrics to KafkaConsumer. The KIP wiki page is below:
> > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > 92+-+Add+per+partition+lag+metrics+to+KafkaConsumer
> > > >
> > > > Thanks,
> > > >
> > > > Jiangjie (Becket) Qin
> > > >
> > >
> >
>

[GitHub] kafka pull request #1868: MINOR: Improve output format of `kafka_reassign_pa...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1868


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #214: MINOR: Make private class FetchManagerMetrics as a ...

[rajatvig]

Github user rajatvig closed the pull request at:

https://github.com/apache/kafka/pull/214


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #219: MINOR: Make SenderMetrics class private with privat...

[rajatvig]

Github user rajatvig closed the pull request at:

https://github.com/apache/kafka/pull/219


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2287: MINOR: Make SenderMetrics class private with priva...

[rajatvig]

GitHub user rajatvig opened a pull request:

https://github.com/apache/kafka/pull/2287

MINOR: Make SenderMetrics class private with private members

to reduce accessibility of the inner class.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rajatvig/kafka fix_sender

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2287.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2287


commit adbc041a9b65fddcefe47fcdd06eaca5cec0a5f0
Author: Rajat Vig 
Date:   2015-09-15T17:27:31Z

Make SenderMetrics class private with private members
to reduce accessibility of the inner class.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: KIP-103: Separation of Internal and External traffic

[Ismael Juma]

Thanks Rajini.

I agree that it's worth thinking about what a fully configurable label
would look like. I'll update the KIP.

Ismael

On 21 Dec 2016 10:53 pm, "Rajini Sivaram"  wrote:

Hi Ismael,

Thank you for the KIP. This is a very useful change.

Once you allow multiple interfaces with the same security protocol, you
will soon also need to be able to configure protocol-specific properties
for each of the interfaces. To use SSL on internal and external networks,
you would almost definitely want different keystores with different
hostname/IP addresses. Similarly for SASL, you might want to enable
different mechanisms, use a different authentication server etc. This is
listed under future work.But it may be worth thinking about what a fully
configurable 'label' looks like. Would every property now become a list/map
like listeners - you would then end up with maps of lists for some
properties. It will good if all properties corresponding to a  label
including listener and advertised.listener are configured consistently - if
that is possible,


On Wed, Dec 21, 2016 at 8:56 PM, Ismael Juma  wrote:

> Hi all,
>
> We've posted "KIP-103: Separation of Internal and External traffic" for
> discussion:
>
> *https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 103%3A+Separation+of+Internal+and+External+traffic
>  103%3A+Separation+of+Internal+and+External+traffic>*
>
> Please take a look. Your feedback is appreciated.
>
> Thanks,
> Ismael
>

Build failed in Jenkins: kafka-trunk-jdk8 #1124

[Apache Jenkins Server]

See 

Changes:

[wangguoz] MINOR: fixup typo in ops.html

--
[...truncated 34375 lines...]
org.apache.kafka.connect.runtime.WorkerTest > testConverterOverrides PASSED

org.apache.kafka.connect.runtime.WorkerSinkTaskThreadedTest > 
testPollsInBackground STARTED

org.apache.kafka.connect.runtime.WorkerSinkTaskThreadedTest > 
testPollsInBackground PASSED

org.apache.kafka.connect.runtime.WorkerSinkTaskThreadedTest > testCommit STARTED

org.apache.kafka.connect.runtime.WorkerSinkTaskThreadedTest > testCommit PASSED

org.apache.kafka.connect.runtime.WorkerSinkTaskThreadedTest > testCommitFailure 
STARTED

org.apache.kafka.connect.runtime.WorkerSinkTaskThreadedTest > testCommitFailure 
PASSED

org.apache.kafka.connect.runtime.WorkerSinkTaskThreadedTest > 
testCommitSuccessFollowedByFailure STARTED

org.apache.kafka.connect.runtime.WorkerSinkTaskThreadedTest > 
testCommitSuccessFollowedByFailure PASSED

org.apache.kafka.connect.runtime.WorkerSinkTaskThreadedTest > 
testCommitConsumerFailure STARTED

org.apache.kafka.connect.runtime.WorkerSinkTaskThreadedTest > 
testCommitConsumerFailure PASSED

org.apache.kafka.connect.runtime.WorkerSinkTaskThreadedTest > testCommitTimeout 
STARTED

org.apache.kafka.connect.runtime.WorkerSinkTaskThreadedTest > testCommitTimeout 
PASSED

org.apache.kafka.connect.runtime.WorkerSinkTaskThreadedTest > 
testAssignmentPauseResume STARTED

org.apache.kafka.connect.runtime.WorkerSinkTaskThreadedTest > 
testAssignmentPauseResume PASSED

org.apache.kafka.connect.runtime.WorkerSinkTaskThreadedTest > testRewind STARTED

org.apache.kafka.connect.runtime.WorkerSinkTaskThreadedTest > testRewind PASSED

org.apache.kafka.connect.runtime.WorkerSinkTaskThreadedTest > 
testRewindOnRebalanceDuringPoll STARTED

org.apache.kafka.connect.runtime.WorkerSinkTaskThreadedTest > 
testRewindOnRebalanceDuringPoll PASSED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorTest > 
testLeaderPerformAssignmentSingleTaskConnectors STARTED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorTest > 
testLeaderPerformAssignmentSingleTaskConnectors PASSED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorTest > 
testNormalJoinGroupFollower STARTED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorTest > 
testNormalJoinGroupFollower PASSED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorTest > 
testMetadata STARTED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorTest > 
testMetadata PASSED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorTest > 
testLeaderPerformAssignment1 STARTED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorTest > 
testLeaderPerformAssignment1 PASSED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorTest > 
testLeaderPerformAssignment2 STARTED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorTest > 
testLeaderPerformAssignment2 PASSED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorTest > 
testJoinLeaderCannotAssign STARTED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorTest > 
testJoinLeaderCannotAssign PASSED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorTest > 
testRejoinGroup STARTED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorTest > 
testRejoinGroup PASSED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorTest > 
testNormalJoinGroupLeader STARTED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorTest > 
testNormalJoinGroupLeader PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testPutConnectorConfig STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testPutConnectorConfig PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testCreateConnector STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testCreateConnector PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testCreateConnectorFailedBasicValidation STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testCreateConnectorFailedBasicValidation PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testCreateConnectorFailedCustomValidation STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testCreateConnectorFailedCustomValidation PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testCreateConnectorAlreadyExists STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testCreateConnectorAlreadyExists PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testDestroyConnector STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testDestroyConnector PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHer

Re: KIP-103: Separation of Internal and External traffic

[Rajini Sivaram]

Hi Ismael,

Thank you for the KIP. This is a very useful change.

Once you allow multiple interfaces with the same security protocol, you
will soon also need to be able to configure protocol-specific properties
for each of the interfaces. To use SSL on internal and external networks,
you would almost definitely want different keystores with different
hostname/IP addresses. Similarly for SASL, you might want to enable
different mechanisms, use a different authentication server etc. This is
listed under future work.But it may be worth thinking about what a fully
configurable 'label' looks like. Would every property now become a list/map
like listeners - you would then end up with maps of lists for some
properties. It will good if all properties corresponding to a  label
including listener and advertised.listener are configured consistently - if
that is possible,


On Wed, Dec 21, 2016 at 8:56 PM, Ismael Juma  wrote:

> Hi all,
>
> We've posted "KIP-103: Separation of Internal and External traffic" for
> discussion:
>
> *https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 103%3A+Separation+of+Internal+and+External+traffic
>  103%3A+Separation+of+Internal+and+External+traffic>*
>
> Please take a look. Your feedback is appreciated.
>
> Thanks,
> Ismael
>

[jira] [Commented] (KAFKA-2170) 10 LogTest cases failed for file.renameTo failed under windows

[Vahid Hashemian (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15768403#comment-15768403
 ] 

Vahid Hashemian commented on KAFKA-2170:


Looks like the PR needs a rebase.

> 10 LogTest cases failed for  file.renameTo failed under windows
> ---
>
> Key: KAFKA-2170
> URL: https://issues.apache.org/jira/browse/KAFKA-2170
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.10.1.0
> Environment: Windows
>Reporter: Honghai Chen
>Assignee: Jay Kreps
>
> get latest code from trunk, then run test 
> gradlew  -i core:test --tests kafka.log.LogTest
> Got 10 cases failed for same reason:
> kafka.common.KafkaStorageException: Failed to change the log file suffix from 
>  to .deleted for log segment 0
>   at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:259)
>   at kafka.log.Log.kafka$log$Log$$asyncDeleteSegment(Log.scala:756)
>   at kafka.log.Log.kafka$log$Log$$deleteSegment(Log.scala:747)
>   at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:514)
>   at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:514)
>   at scala.collection.immutable.List.foreach(List.scala:318)
>   at kafka.log.Log.deleteOldSegments(Log.scala:514)
>   at kafka.log.LogTest.testAsyncDelete(LogTest.scala:633)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:601)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:44)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:180)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:41)
>   at org.junit.runners.ParentRunner$1.evaluate(ParentRunner.java:173)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:220)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:86)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:49)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:69)
>   at 
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:48)
>   at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:601)
>   at 
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
>   at 
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
>   at 
> org.gradle.messaging.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
>   at 
> org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
>   at $Proxy2.processTestClass(Unknown Source)
>   at 
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:105)
>   at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:601)
>   at 
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
>   at 
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
>   at 
> org.gradle.messaging.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:355)
>  

[jira] [Updated] (KAFKA-3284) Consider removing beta label in security documentation

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3284:
---
Status: Patch Available  (was: Open)

> Consider removing beta label in security documentation
> --
>
> Key: KAFKA-3284
> URL: https://issues.apache.org/jira/browse/KAFKA-3284
> Project: Kafka
>  Issue Type: Improvement
>  Components: security
>Reporter: Ismael Juma
>Assignee: Ismael Juma
> Fix For: 0.10.2.0
>
>
> We currently state that our security support is beta. It would be good to 
> remove that for 0.10.0.0.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3284) Consider removing beta label in security documentation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15768357#comment-15768357
 ] 

ASF GitHub Bot commented on KAFKA-3284:
---

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/2286

KAFKA-3284: Remove beta label from security documentation



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka kafka-3284-security-beta-label

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2286.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2286


commit 4b734f7ecd742a6d05efce91c12ec6f0c4ca64ab
Author: Ismael Juma 
Date:   2016-12-21T22:29:45Z

KAFKA-3284: Remove beta label from security documentation




> Consider removing beta label in security documentation
> --
>
> Key: KAFKA-3284
> URL: https://issues.apache.org/jira/browse/KAFKA-3284
> Project: Kafka
>  Issue Type: Improvement
>  Components: security
>Reporter: Ismael Juma
>Assignee: Ismael Juma
> Fix For: 0.10.2.0
>
>
> We currently state that our security support is beta. It would be good to 
> remove that for 0.10.0.0.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2286: KAFKA-3284: Remove beta label from security docume...

[ijuma]

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/2286

KAFKA-3284: Remove beta label from security documentation



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka kafka-3284-security-beta-label

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2286.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2286


commit 4b734f7ecd742a6d05efce91c12ec6f0c4ca64ab
Author: Ismael Juma 
Date:   2016-12-21T22:29:45Z

KAFKA-3284: Remove beta label from security documentation




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-2980) ZookeeperConsumerConnector may enter deadlock if a rebalance occurs during a stream creation.

[Jiangjie Qin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jiangjie Qin updated KAFKA-2980:

Resolution: Cannot Reproduce
Status: Resolved  (was: Patch Available)

> ZookeeperConsumerConnector may enter deadlock if a rebalance occurs during a 
> stream creation.
> -
>
> Key: KAFKA-2980
> URL: https://issues.apache.org/jira/browse/KAFKA-2980
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>
> The following sequence caused problems:
> 1. Multiple ZookeeperConsumerConnector in the same group start at the same 
> time.
> 2. The user consumer thread called createMessageStreamsByFilter()
> 3. Right before the user consumer thread enters syncedRebalance(), a 
> rebalance was triggered by another consumer joining the group.
> 4. Because the watcher executor has been up and running at this point, the 
> executor watcher will start to rebalance. Now both the user consumer thread 
> and the executor watcher are trying to rebalance.
> 5. The executor watcher wins this time. It finishes the rebalance, so the 
> fetchers started to run.
> 6. After that the user consumer thread will try to rebalance again, but it 
> blocks when trying to stop the fetchers. Since the fetcher threads are 
> blocked on putting data chunk into data chunk queue.
> 7. In this case, because there is no thread taking messages out of data chunk 
> queue, the fetcher thread will not be able to make process. Neither does the 
> user consumer thread. So we have a deadlock here.
> The current code works if there is no fetcher thread running when 
> createMessageStreams/createMessageStreamsByFilter is called. The simple fix 
> is to let those two methods acquire the rebalance lock.
> Although it is a fix to old consumer, but since the fix is quite small and 
> important for people who are still using old consumer. I think it still worth 
> doing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2980) ZookeeperConsumerConnector may enter deadlock if a rebalance occurs during a stream creation.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15768278#comment-15768278
 ] 

ASF GitHub Bot commented on KAFKA-2980:
---

Github user becketqin closed the pull request at:

https://github.com/apache/kafka/pull/660


> ZookeeperConsumerConnector may enter deadlock if a rebalance occurs during a 
> stream creation.
> -
>
> Key: KAFKA-2980
> URL: https://issues.apache.org/jira/browse/KAFKA-2980
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>
> The following sequence caused problems:
> 1. Multiple ZookeeperConsumerConnector in the same group start at the same 
> time.
> 2. The user consumer thread called createMessageStreamsByFilter()
> 3. Right before the user consumer thread enters syncedRebalance(), a 
> rebalance was triggered by another consumer joining the group.
> 4. Because the watcher executor has been up and running at this point, the 
> executor watcher will start to rebalance. Now both the user consumer thread 
> and the executor watcher are trying to rebalance.
> 5. The executor watcher wins this time. It finishes the rebalance, so the 
> fetchers started to run.
> 6. After that the user consumer thread will try to rebalance again, but it 
> blocks when trying to stop the fetchers. Since the fetcher threads are 
> blocked on putting data chunk into data chunk queue.
> 7. In this case, because there is no thread taking messages out of data chunk 
> queue, the fetcher thread will not be able to make process. Neither does the 
> user consumer thread. So we have a deadlock here.
> The current code works if there is no fetcher thread running when 
> createMessageStreams/createMessageStreamsByFilter is called. The simple fix 
> is to let those two methods acquire the rebalance lock.
> Although it is a fix to old consumer, but since the fix is quite small and 
> important for people who are still using old consumer. I think it still worth 
> doing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #660: KAFKA-2980 Fix deadlock when ZookeeperConsumerConne...

[becketqin]

Github user becketqin closed the pull request at:

https://github.com/apache/kafka/pull/660


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [VOTE] KIP-92 - Add per partition lag metrics to KafkaConsumer

[Ismael Juma]

That seems reasonable. It would be good to update the KIP to make this
clear.

Ismael

On Wed, Dec 21, 2016 at 1:49 PM, Becket Qin  wrote:

> Hi Ismael,
>
> Thanks for the comments. Good observation. I guess for max lag of all the
> partitions the average value is less meaningful because the lag can be from
> different partitions, so an average of lags from different partitions at
> different times seems hard to reason about. On the other hand, for per
> partition lag, the average value of the same partition at different times
> seems having some real meaning?
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Wed, Dec 21, 2016 at 1:02 PM, Ismael Juma  wrote:
>
> > Thanks for the KIP, it's a useful improvement. Just one question, the KIP
> > states that it's about adding per partition metrics for metrics that
> > already exist. However, when I look at the code, it seems that
> > `records-lag-avg` doesn't exist?
> >
> > this.recordsFetchLag = metrics.sensor("records-lag");
> > this.recordsFetchLag.add(metrics.metricName("records-lag-max",
> > this.metricGrpName, "The maximum lag in terms of number of records for
> any
> > partition in this window"), new Max());
> >
> > Ismael
> >
> > On Thu, Dec 15, 2016 at 5:32 PM, Becket Qin 
> wrote:
> >
> > > Hi,
> > >
> > > I want to start a voting thread on KIP-92 which proposes to add per
> > > partition lag metrics to KafkaConsumer. The KIP wiki page is below:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 92+-+Add+per+partition+lag+metrics+to+KafkaConsumer
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > >
> >
>

Re: [VOTE] KIP-92 - Add per partition lag metrics to KafkaConsumer

[Becket Qin]

Hi Ismael,

Thanks for the comments. Good observation. I guess for max lag of all the
partitions the average value is less meaningful because the lag can be from
different partitions, so an average of lags from different partitions at
different times seems hard to reason about. On the other hand, for per
partition lag, the average value of the same partition at different times
seems having some real meaning?

Thanks,

Jiangjie (Becket) Qin

On Wed, Dec 21, 2016 at 1:02 PM, Ismael Juma  wrote:

> Thanks for the KIP, it's a useful improvement. Just one question, the KIP
> states that it's about adding per partition metrics for metrics that
> already exist. However, when I look at the code, it seems that
> `records-lag-avg` doesn't exist?
>
> this.recordsFetchLag = metrics.sensor("records-lag");
> this.recordsFetchLag.add(metrics.metricName("records-lag-max",
> this.metricGrpName, "The maximum lag in terms of number of records for any
> partition in this window"), new Max());
>
> Ismael
>
> On Thu, Dec 15, 2016 at 5:32 PM, Becket Qin  wrote:
>
> > Hi,
> >
> > I want to start a voting thread on KIP-92 which proposes to add per
> > partition lag metrics to KafkaConsumer. The KIP wiki page is below:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 92+-+Add+per+partition+lag+metrics+to+KafkaConsumer
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
>

[GitHub] kafka pull request #2077: MINOR: Fix document header/footer links

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2077


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-4565) Separation of Internal and External traffic

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-4565:
---
Summary: Separation of Internal and External traffic  (was: Separate 
Internal and External traffic)

> Separation of Internal and External traffic
> ---
>
> Key: KAFKA-4565
> URL: https://issues.apache.org/jira/browse/KAFKA-4565
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ismael Juma
>Assignee: Ismael Juma
>
> During the 0.9.0.0 release cycle, support for multiple listeners per broker 
> was introduced (KAFKA-1809). Each listener is associated with a security 
> protocol, ip/host and port. When combined with the advertised listeners 
> mechanism, there is a fair amount of flexibility with one limitation: at most 
> one listener per security protocol in each of the two configs (listeners and 
> advertised.listeners).
> In some environments, one may want to differentiate between external clients, 
> internal clients and replication traffic independently of the security 
> protocol for cost, performance and security reasons. See the KIP for more 
> details: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-103%3A+Separate+Internal+and+External+traffic



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4565) Separation of Internal and External traffic

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-4565:
---
Description: 
During the 0.9.0.0 release cycle, support for multiple listeners per broker was 
introduced (KAFKA-1809). Each listener is associated with a security protocol, 
ip/host and port. When combined with the advertised listeners mechanism, there 
is a fair amount of flexibility with one limitation: at most one listener per 
security protocol in each of the two configs (listeners and 
advertised.listeners).

In some environments, one may want to differentiate between external clients, 
internal clients and replication traffic independently of the security protocol 
for cost, performance and security reasons. See the KIP for more details: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-103%3A+Separation+of+Internal+and+External+traffic

  was:
During the 0.9.0.0 release cycle, support for multiple listeners per broker was 
introduced (KAFKA-1809). Each listener is associated with a security protocol, 
ip/host and port. When combined with the advertised listeners mechanism, there 
is a fair amount of flexibility with one limitation: at most one listener per 
security protocol in each of the two configs (listeners and 
advertised.listeners).

In some environments, one may want to differentiate between external clients, 
internal clients and replication traffic independently of the security protocol 
for cost, performance and security reasons. See the KIP for more details: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-103%3A+Separate+Internal+and+External+traffic


> Separation of Internal and External traffic
> ---
>
> Key: KAFKA-4565
> URL: https://issues.apache.org/jira/browse/KAFKA-4565
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ismael Juma
>Assignee: Ismael Juma
>
> During the 0.9.0.0 release cycle, support for multiple listeners per broker 
> was introduced (KAFKA-1809). Each listener is associated with a security 
> protocol, ip/host and port. When combined with the advertised listeners 
> mechanism, there is a fair amount of flexibility with one limitation: at most 
> one listener per security protocol in each of the two configs (listeners and 
> advertised.listeners).
> In some environments, one may want to differentiate between external clients, 
> internal clients and replication traffic independently of the security 
> protocol for cost, performance and security reasons. See the KIP for more 
> details: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-103%3A+Separation+of+Internal+and+External+traffic



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2260) Allow specifying expected offset on produce

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15768183#comment-15768183
 ] 

Ismael Juma commented on KAFKA-2260:


The following KIP may be of interest: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging

> Allow specifying expected offset on produce
> ---
>
> Key: KAFKA-2260
> URL: https://issues.apache.org/jira/browse/KAFKA-2260
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Ben Kirwin
>Assignee: Ewen Cheslack-Postava
>Priority: Minor
> Attachments: KAFKA-2260.patch, expected-offsets.patch
>
>
> I'd like to propose a change that adds a simple CAS-like mechanism to the 
> Kafka producer. This update has a small footprint, but enables a bunch of 
> interesting uses in stream processing or as a commit log for process state.
> h4. Proposed Change
> In short:
> - Allow the user to attach a specific offset to each message produced.
> - The server assigns offsets to messages in the usual way. However, if the 
> expected offset doesn't match the actual offset, the server should fail the 
> produce request instead of completing the write.
> This is a form of optimistic concurrency control, like the ubiquitous 
> check-and-set -- but instead of checking the current value of some state, it 
> checks the current offset of the log.
> h4. Motivation
> Much like check-and-set, this feature is only useful when there's very low 
> contention. Happily, when Kafka is used as a commit log or as a 
> stream-processing transport, it's common to have just one producer (or a 
> small number) for a given partition -- and in many of these cases, predicting 
> offsets turns out to be quite useful.
> - We get the same benefits as the 'idempotent producer' proposal: a producer 
> can retry a write indefinitely and be sure that at most one of those attempts 
> will succeed; and if two producers accidentally write to the end of the 
> partition at once, we can be certain that at least one of them will fail.
> - It's possible to 'bulk load' Kafka this way -- you can write a list of n 
> messages consecutively to a partition, even if the list is much larger than 
> the buffer size or the producer has to be restarted.
> - If a process is using Kafka as a commit log -- reading from a partition to 
> bootstrap, then writing any updates to that same partition -- it can be sure 
> that it's seen all of the messages in that partition at the moment it does 
> its first (successful) write.
> There's a bunch of other similar use-cases here, but they all have roughly 
> the same flavour.
> h4. Implementation
> The major advantage of this proposal over other suggested transaction / 
> idempotency mechanisms is its minimality: it gives the 'obvious' meaning to a 
> currently-unused field, adds no new APIs, and requires very little new code 
> or additional work from the server.
> - Produced messages already carry an offset field, which is currently ignored 
> by the server. This field could be used for the 'expected offset', with a 
> sigil value for the current behaviour. (-1 is a natural choice, since it's 
> already used to mean 'next available offset'.)
> - We'd need a new error and error code for a 'CAS failure'.
> - The server assigns offsets to produced messages in 
> {{ByteBufferMessageSet.validateMessagesAndAssignOffsets}}. After this 
> changed, this method would assign offsets in the same way -- but if they 
> don't match the offset in the message, we'd return an error instead of 
> completing the write.
> - To avoid breaking existing clients, this behaviour would need to live 
> behind some config flag. (Possibly global, but probably more useful 
> per-topic?)
> I understand all this is unsolicited and possibly strange: happy to answer 
> questions, and if this seems interesting, I'd be glad to flesh this out into 
> a full KIP or patch. (And apologies if this is the wrong venue for this sort 
> of thing!)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2260) Allow specifying expected offset on produce

[Enrico Olivelli (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15768180#comment-15768180
 ] 

Enrico Olivelli commented on KAFKA-2260:


I am interested in this new feature as it will enable kafka to be used as 
transaction log for replicated state machines where it is important that only 
one node can actually modify the state of the system, that it to write to the 
log

> Allow specifying expected offset on produce
> ---
>
> Key: KAFKA-2260
> URL: https://issues.apache.org/jira/browse/KAFKA-2260
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Ben Kirwin
>Assignee: Ewen Cheslack-Postava
>Priority: Minor
> Attachments: KAFKA-2260.patch, expected-offsets.patch
>
>
> I'd like to propose a change that adds a simple CAS-like mechanism to the 
> Kafka producer. This update has a small footprint, but enables a bunch of 
> interesting uses in stream processing or as a commit log for process state.
> h4. Proposed Change
> In short:
> - Allow the user to attach a specific offset to each message produced.
> - The server assigns offsets to messages in the usual way. However, if the 
> expected offset doesn't match the actual offset, the server should fail the 
> produce request instead of completing the write.
> This is a form of optimistic concurrency control, like the ubiquitous 
> check-and-set -- but instead of checking the current value of some state, it 
> checks the current offset of the log.
> h4. Motivation
> Much like check-and-set, this feature is only useful when there's very low 
> contention. Happily, when Kafka is used as a commit log or as a 
> stream-processing transport, it's common to have just one producer (or a 
> small number) for a given partition -- and in many of these cases, predicting 
> offsets turns out to be quite useful.
> - We get the same benefits as the 'idempotent producer' proposal: a producer 
> can retry a write indefinitely and be sure that at most one of those attempts 
> will succeed; and if two producers accidentally write to the end of the 
> partition at once, we can be certain that at least one of them will fail.
> - It's possible to 'bulk load' Kafka this way -- you can write a list of n 
> messages consecutively to a partition, even if the list is much larger than 
> the buffer size or the producer has to be restarted.
> - If a process is using Kafka as a commit log -- reading from a partition to 
> bootstrap, then writing any updates to that same partition -- it can be sure 
> that it's seen all of the messages in that partition at the moment it does 
> its first (successful) write.
> There's a bunch of other similar use-cases here, but they all have roughly 
> the same flavour.
> h4. Implementation
> The major advantage of this proposal over other suggested transaction / 
> idempotency mechanisms is its minimality: it gives the 'obvious' meaning to a 
> currently-unused field, adds no new APIs, and requires very little new code 
> or additional work from the server.
> - Produced messages already carry an offset field, which is currently ignored 
> by the server. This field could be used for the 'expected offset', with a 
> sigil value for the current behaviour. (-1 is a natural choice, since it's 
> already used to mean 'next available offset'.)
> - We'd need a new error and error code for a 'CAS failure'.
> - The server assigns offsets to produced messages in 
> {{ByteBufferMessageSet.validateMessagesAndAssignOffsets}}. After this 
> changed, this method would assign offsets in the same way -- but if they 
> don't match the offset in the message, we'd return an error instead of 
> completing the write.
> - To avoid breaking existing clients, this behaviour would need to live 
> behind some config flag. (Possibly global, but probably more useful 
> per-topic?)
> I understand all this is unsolicited and possibly strange: happy to answer 
> questions, and if this seems interesting, I'd be glad to flesh this out into 
> a full KIP or patch. (And apologies if this is the wrong venue for this sort 
> of thing!)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] KIP-92 - Add per partition lag metrics to KafkaConsumer

[Ismael Juma]

Thanks for the KIP, it's a useful improvement. Just one question, the KIP
states that it's about adding per partition metrics for metrics that
already exist. However, when I look at the code, it seems that
`records-lag-avg` doesn't exist?

this.recordsFetchLag = metrics.sensor("records-lag");
this.recordsFetchLag.add(metrics.metricName("records-lag-max",
this.metricGrpName, "The maximum lag in terms of number of records for any
partition in this window"), new Max());

Ismael

On Thu, Dec 15, 2016 at 5:32 PM, Becket Qin  wrote:

> Hi,
>
> I want to start a voting thread on KIP-92 which proposes to add per
> partition lag metrics to KafkaConsumer. The KIP wiki page is below:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 92+-+Add+per+partition+lag+metrics+to+KafkaConsumer
>
> Thanks,
>
> Jiangjie (Becket) Qin
>

KIP-103: Separation of Internal and External traffic

[Ismael Juma]

Hi all,

We've posted "KIP-103: Separation of Internal and External traffic" for
discussion:

*https://cwiki.apache.org/confluence/display/KAFKA/KIP-103%3A+Separation+of+Internal+and+External+traffic
*

Please take a look. Your feedback is appreciated.

Thanks,
Ismael

[GitHub] kafka pull request #1958: MINOR: trivial doc cleanup

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1958


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to normal : kafka-trunk-jdk8 #1123

[Apache Jenkins Server]

See 

[jira] [Commented] (KAFKA-2260) Allow specifying expected offset on produce

[Bill Warshaw (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15768086#comment-15768086
 ] 

Bill Warshaw commented on KAFKA-2260:
-

I'd like to revive the discussion on this KIP.  I've worked on multiple 
projects where this functionality would have saved us a lot of effort and 
complexity.

I have a working proof-of-concept branch locally.  I had to switch approaches 
from the attached patch, because of changes to how Kafka uses the offset field 
in KIP-31 
(https://cwiki.apache.org/confluence/display/KAFKA/KIP-31+-+Move+to+relative+offsets+in+compressed+message+sets).
  I ended up adding a field to the {{TOPIC_PRODUCE_DATA}} API; the changes are 
fairly minor, and are mostly just updating method signatures.

I'd be interested in contributing a patch upstream if there's still a desire 
for it.

> Allow specifying expected offset on produce
> ---
>
> Key: KAFKA-2260
> URL: https://issues.apache.org/jira/browse/KAFKA-2260
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Ben Kirwin
>Assignee: Ewen Cheslack-Postava
>Priority: Minor
> Attachments: KAFKA-2260.patch, expected-offsets.patch
>
>
> I'd like to propose a change that adds a simple CAS-like mechanism to the 
> Kafka producer. This update has a small footprint, but enables a bunch of 
> interesting uses in stream processing or as a commit log for process state.
> h4. Proposed Change
> In short:
> - Allow the user to attach a specific offset to each message produced.
> - The server assigns offsets to messages in the usual way. However, if the 
> expected offset doesn't match the actual offset, the server should fail the 
> produce request instead of completing the write.
> This is a form of optimistic concurrency control, like the ubiquitous 
> check-and-set -- but instead of checking the current value of some state, it 
> checks the current offset of the log.
> h4. Motivation
> Much like check-and-set, this feature is only useful when there's very low 
> contention. Happily, when Kafka is used as a commit log or as a 
> stream-processing transport, it's common to have just one producer (or a 
> small number) for a given partition -- and in many of these cases, predicting 
> offsets turns out to be quite useful.
> - We get the same benefits as the 'idempotent producer' proposal: a producer 
> can retry a write indefinitely and be sure that at most one of those attempts 
> will succeed; and if two producers accidentally write to the end of the 
> partition at once, we can be certain that at least one of them will fail.
> - It's possible to 'bulk load' Kafka this way -- you can write a list of n 
> messages consecutively to a partition, even if the list is much larger than 
> the buffer size or the producer has to be restarted.
> - If a process is using Kafka as a commit log -- reading from a partition to 
> bootstrap, then writing any updates to that same partition -- it can be sure 
> that it's seen all of the messages in that partition at the moment it does 
> its first (successful) write.
> There's a bunch of other similar use-cases here, but they all have roughly 
> the same flavour.
> h4. Implementation
> The major advantage of this proposal over other suggested transaction / 
> idempotency mechanisms is its minimality: it gives the 'obvious' meaning to a 
> currently-unused field, adds no new APIs, and requires very little new code 
> or additional work from the server.
> - Produced messages already carry an offset field, which is currently ignored 
> by the server. This field could be used for the 'expected offset', with a 
> sigil value for the current behaviour. (-1 is a natural choice, since it's 
> already used to mean 'next available offset'.)
> - We'd need a new error and error code for a 'CAS failure'.
> - The server assigns offsets to produced messages in 
> {{ByteBufferMessageSet.validateMessagesAndAssignOffsets}}. After this 
> changed, this method would assign offsets in the same way -- but if they 
> don't match the offset in the message, we'd return an error instead of 
> completing the write.
> - To avoid breaking existing clients, this behaviour would need to live 
> behind some config flag. (Possibly global, but probably more useful 
> per-topic?)
> I understand all this is unsolicited and possibly strange: happy to answer 
> questions, and if this seems interesting, I'd be glad to flesh this out into 
> a full KIP or patch. (And apologies if this is the wrong venue for this sort 
> of thing!)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3587) LogCleaner fails due to incorrect offset map computation on a replica

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-3587:
-
Fix Version/s: 0.9.0.2

> LogCleaner fails due to incorrect offset map computation on a replica
> -
>
> Key: KAFKA-3587
> URL: https://issues.apache.org/jira/browse/KAFKA-3587
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.1
> Environment: Linux
>Reporter: Kiran Pillarisetty
>Assignee: Edoardo Comar
> Fix For: 0.10.0.0, 0.9.0.2
>
> Attachments: 0001-POC-improving-deduping-segments.patch
>
>
> Log Cleaner fails to compact a segment even when the number of messages in it 
> is less than the offset map.
> In version 0.9.0.1, (LogCleaner.scala -> buildOffsetMap()), LogCleaner 
> computes segment size by subtracting segment's base offset from the latest 
> offset ("segmentSize = segment.nextOffset() - segment.baseOffset").  This 
> works fine until you create another replica. When you create a replica, it's 
> segment could contain data which is already compacted on other brokers. 
> Depending up on the type of data, offset difference could be too big, larger 
> than the offset map (maxDesiredMapSize), and that causes LogCleaner to fail 
> on that segment.
> Scenario:
> - Kafka 0.9.0.1
> - Cluster has two brokers.
> - Server.properties:
> log.cleaner.enable=true
> log.cleaner.dedupe.buffer.size=10485760 #10MB
> log.roll.ms=30
> delete.topic.enable=true
> log.cleanup.policy=compact
> Steps to reproduce:
> 1. Create a topic with replication-factor of 1.
> ./kafka-topics.sh --zookeeper=localhost:2181 --create --topic 
> test.log.compact.1M --partitions 1 --replication-factor 1 --config 
> cleanup.policy=compact --config segment.ms=30
> 2. Use kafka-console-producer.sh to produce a single message with the 
> following key:
> LC1,{"test": "xyz"}
> 3. Use  kafka-console-producer.sh to produce a large number of messages with 
> the following key:
> LC2,{"test": "abc"}
> 4. Let log cleaner run. Make sure log is compacted.  Verify with:
>  ./kafka-run-class.sh kafka.tools.DumpLogSegments  --files 
> .log  --print-data-log
> Dumping .log
> Starting offset: 0
> offset: 0 position: 0 isvalid: true payloadsize: 11 magic: 0 compresscodec: 
> NoCompressionCodec crc: 3067045277 keysize: 11 key: LC1 payload: {"test": 
> "xyz"}
> offset: 7869818 position: 48 isvalid: true payloadsize: 11 magic: 0 
> compresscodec: NoCompressionCodec crc: 2668089711 keysize: 11 key: LC2 
> payload: {"test": "abc"}
> 5.  Increase Replication Factor to 2.  Followed these steps: 
> http://kafka.apache.org/documentation.html#basic_ops_increase_replication_factor
> 6. Notice that log cleaner fails to compact the newly created replica with 
> the following error.
> [2016-04-18 14:49:45,599] ERROR [kafka-log-cleaner-thread-0], Error due to  
> (kafka.log.LogCleaner)
> java.lang.IllegalArgumentException: requirement failed: 7206179 messages in 
> segment test.log.compact.1M-0/.log but offset map can fit 
> only 393215. You can increase log.cleaner.dedupe.buffer.size or decrease 
> log.cleaner.threads
> at scala.Predef$.require(Predef.scala:219)
> at 
> kafka.log.Cleaner$$anonfun$buildOffsetMap$4.apply(LogCleaner.scala:584)
> at 
> kafka.log.Cleaner$$anonfun$buildOffsetMap$4.apply(LogCleaner.scala:580)
> at 
> scala.collection.immutable.Stream$StreamWithFilter.foreach(Stream.scala:570)
> at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:580)
> at kafka.log.Cleaner.clean(LogCleaner.scala:322)
> at 
> kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:230)
> at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:208)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
> [2016-04-18 14:49:45,601] INFO [kafka-log-cleaner-thread-0], Stopped  
> (kafka.log.LogCleaner)
> 7. Examine the entries in the replica segment:
> ./kafka-run-class.sh kafka.tools.DumpLogSegments --files 
> .log  --print-data-log
> There are only 218418 messages in that segment.
> However, Log Cleaner seems to think that there are 7206179 messages in that 
> segment (as per the above error)
> Error stems from this line in LogCleaner.scala:
> """val segmentSize = segment.nextOffset() - segment.baseOffset"""
> In Replica's log segment file ( .log), ending offset is 
> 7206178. Beginning offset is 0.  That makes Log Cleaner think that there are 
> 7206179 messages in that segment although there are only 218418 messages in 
> it.
> IMO,  to address this kind of scenario, LogCleaner.scala should check for the 
> number of messages in the segment, instead of subtracting beginning offset 
> from the

[GitHub] kafka pull request #2269: fixup typo

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2269


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-3564) Count metric always increments by 1.0

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-3564:
-
Resolution: Not A Bug
Status: Resolved  (was: Patch Available)

Closing as not a bug for now. Please feel free to re-open if there are any 
follow-up questions.

> Count metric always increments by 1.0
> -
>
> Key: KAFKA-3564
> URL: https://issues.apache.org/jira/browse/KAFKA-3564
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.1.0
>Reporter: Michael Coon
>Assignee: Kim Christensen
>
> The Count metric's update method always increments its value by 1.0 instead 
> of the value passed to it. If this is by design, it's misleading as I want to 
> be able to count based on values I send to the record method.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] 0.10.1.1 RC1

[Gwen Shapira]

Thanks for everyone who voted. Having community vote of confidence in
bugfix releases is hugely important, especially as we are trying to
support more in-flight versions :)

On Wed, Dec 21, 2016 at 8:49 AM, Ismael Juma  wrote:
> Hi Guozhang,
>
> Thanks for driving this release as it includes a few important fixes. I
> belatedly tested Scala 2.12 binaries using the quickstart,  +1
> (non-binding) from me.
>
> Ismael
>
> On Thu, Dec 15, 2016 at 1:29 PM, Guozhang Wang  wrote:
>
>> Hello Kafka users, developers and client-developers,
>>
>> This is the second, and hopefully the last candidate for the release of
>> Apache Kafka 0.10.1.1 before the break. This is a bug fix release and it
>> includes fixes and improvements from 30 JIRAs. See the release notes for
>> more details:
>>
>> http://home.apache.org/~guozhang/kafka-0.10.1.1-rc1/RELEASE_NOTES.html
>>
>> *** Please download, test and vote by Tuesday, 20 December, 8pm PT ***
>>
>> Kafka's KEYS file containing PGP keys we use to sign the release:
>> http://kafka.apache.org/KEYS
>>
>> * Release artifacts to be voted upon (source and binary):
>> http://home.apache.org/~guozhang/kafka-0.10.1.1-rc1/
>>
>> * Maven artifacts to be voted upon:
>> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>>
>> NOTE the artifacts include the ones built from Scala 2.12.1 and Java8,
>> which are treated a pre-alpha artifacts for the Scala community to try and
>> test it out:
>>
>> https://repository.apache.org/content/groups/staging/org/apa
>> che/kafka/kafka_2.12/0.10.1.1/
>>
>> We will formally add the scala 2.12 support in future minor releases.
>>
>>
>> * Javadoc:
>> http://home.apache.org/~guozhang/kafka-0.10.1.1-rc1/javadoc/
>>
>> * Tag to be voted upon (off 0.10.0 branch) is the 0.10.0.1-rc0 tag:
>> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=
>> c3638376708ee6c02dfe4e57747acae0126fa6e7
>>
>>
>> Thanks,
>> Guozhang
>>
>> --
>> -- Guozhang
>>



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

[Apurva Mehta]

Hi Joel,

The alternatives are embedded in the 'discussion' sections which are spread
throughout the google doc.

Admittedly, we have not covered high level alternatives like those which
have been brought up in this thread. In particular, having a separate log
for transactional mesages and also having multiple producers participate in
a single transaction.

This is an omission which we will correct.

Thanks,
Apurva

On Wed, Dec 21, 2016 at 10:34 AM, Joel Koshy  wrote:

> >
> >
> > @Joel,
> >
> > I read over your wiki, and apart from the introduction of the notion of
> > journal partitions --whose pros and cons are already being discussed--
> you
> > also introduce the notion of a 'producer group' which enables multiple
> > producers to participate in a single transaction. This is completely
> > opposite of the model in the KIP where a transaction is defined by a
> > producer id, and hence there is a 1-1 mapping between producers and
> > transactions. Further, each producer can have exactly one in-flight
> > transaction at a time in the KIP.
> >
>
> Hi Apurva - yes I did notice those differences among other things :) BTW, I
> haven't yet gone through the google-doc carefully but on a skim it does not
> seem to contain any rejected alternatives as the wiki states.
>

Re: [VOTE] KIP-92 - Add per partition lag metrics to KafkaConsumer

[Guozhang Wang]

+1, thanks.

On Wed, Dec 21, 2016 at 10:26 AM, radai  wrote:

> +1
>
> On Wed, Dec 21, 2016 at 9:51 AM, Dong Lin  wrote:
>
> > +1 (non-binding)
> >
> > On Thu, Dec 15, 2016 at 5:32 PM, Becket Qin 
> wrote:
> >
> > > Hi,
> > >
> > > I want to start a voting thread on KIP-92 which proposes to add per
> > > partition lag metrics to KafkaConsumer. The KIP wiki page is below:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 92+-+Add+per+partition+lag+metrics+to+KafkaConsumer
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > >
> >
>



-- 
-- Guozhang

Re: [VOTE] KIP-92 - Add per partition lag metrics to KafkaConsumer

[Joel Koshy]

+1

On Wed, Dec 21, 2016 at 10:26 AM, radai  wrote:

> +1
>
> On Wed, Dec 21, 2016 at 9:51 AM, Dong Lin  wrote:
>
> > +1 (non-binding)
> >
> > On Thu, Dec 15, 2016 at 5:32 PM, Becket Qin 
> wrote:
> >
> > > Hi,
> > >
> > > I want to start a voting thread on KIP-92 which proposes to add per
> > > partition lag metrics to KafkaConsumer. The KIP wiki page is below:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 92+-+Add+per+partition+lag+metrics+to+KafkaConsumer
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > >
> >
>

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

[Joel Koshy]

>
>
> @Joel,
>
> I read over your wiki, and apart from the introduction of the notion of
> journal partitions --whose pros and cons are already being discussed-- you
> also introduce the notion of a 'producer group' which enables multiple
> producers to participate in a single transaction. This is completely
> opposite of the model in the KIP where a transaction is defined by a
> producer id, and hence there is a 1-1 mapping between producers and
> transactions. Further, each producer can have exactly one in-flight
> transaction at a time in the KIP.
>

Hi Apurva - yes I did notice those differences among other things :) BTW, I
haven't yet gone through the google-doc carefully but on a skim it does not
seem to contain any rejected alternatives as the wiki states.

[jira] [Updated] (KAFKA-2620) Introduce Scalariform

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-2620:
-
Reviewer: Gwen Shapira

> Introduce Scalariform
> -
>
> Key: KAFKA-2620
> URL: https://issues.apache.org/jira/browse/KAFKA-2620
> Project: Kafka
>  Issue Type: Bug
>  Components: build
>Reporter: Grant Henke
>Assignee: Grant Henke
>
> Many of our reviews include nit comments related to Scala style. Adding 
> [Scalariform|https://github.com/daniel-trinh/scalariform] allows us to 
> reformat the code based on configurable standards at build time, ensuring 
> uniform readability and a short review/commit cycle. 
> I expect this will have some discussion around the rules we would like to 
> include, and if we actually want to adopt this. I will submit a sample patch 
> to start the discussion.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-92 - Add per partition lag metrics to KafkaConsumer

[Becket Qin]

Good point. I think it makes sense to remove the corresponding metric if
the consumer does not own a partition anymore. I have updated the KIP.

On Wed, Dec 21, 2016 at 10:24 AM, Joel Koshy  wrote:

> LGTM. However, can you comment on the effect of releasing ownership of
> partitions after a rebalance? For e.g., should it reset itself to (say) -1?
> or removed? This really applies to any per-partition metrics that we intend
> to maintain in the consumer.
>
> On Mon, Nov 14, 2016 at 9:35 AM, Becket Qin  wrote:
>
> > Hey Michael,
> >
> > Thanks for the comments. Exposing the lag on the client side may serve
> some
> > cases a little different from monitoring. For example, one of the use
> case
> > we have was that an application has some high priority and low priority
> > topics to consume. They want to switch between consuming from high
> priority
> > and low priority topics alternately based on the lag on the partition.
> i.e.
> > if the consume has already caught up with the log end offset of a high
> > priority topic, the application will switch to consume from the low
> > priority topics. Otherwise it will continue consuming from the high
> > priority topics. We have seen a few other similar use cases that require
> a
> > programmatic access to the lag. Although people can always use
> > offsetsForTimes() to get the LEO, but it is more expensive call involving
> > an RPC and is a blocking call.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> >
> >
> > On Mon, Nov 14, 2016 at 9:18 AM, Michael Pearce 
> > wrote:
> >
> > > Should state I have no objections adding this client side, just more a
> > > question to why we don't look and propose to add this broker side also.
> > >
> > > Sent using OWA for iPhone
> > > 
> > > From: Michael Pearce 
> > > Sent: Monday, November 14, 2016 4:58:45 PM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-92 - Add per partition lag metrics to
> > > KafkaConsumer
> > >
> > > Why do we not look to expose the lag broker side centrally?
> > >
> > > Eg like burrow.
> > >
> > > From an operations point it's a lot easier to monitor lag centrally
> than
> > > per application. Also then you'd be able to see lag of consumers not
> > alive
> > > or stalled.
> > >
> > > The information if the consumer uses Kafka based or zookeeper offsets
> is
> > > available to the broker.
> > > 
> > > From: Becket Qin 
> > > Sent: Sunday, November 13, 2016 4:13:01 AM
> > > To: dev@kafka.apache.org
> > > Subject: [DISCUSS] KIP-92 - Add per partition lag metrics to
> > KafkaConsumer
> > >
> > > Hi,
> > >
> > > We created KIP-92 to propose adding per partition lag metrics to
> > > KafkaConsumer.
> > >
> > > The KIP wiki link is the following:
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 92+-+Add+per+partition+lag+metrics+to+KafkaConsumer
> > >
> > > Comments are welcome.
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > > The information contained in this email is strictly confidential and
> for
> > > the use of the addressee only, unless otherwise indicated. If you are
> not
> > > the intended recipient, please do not read, copy, use or disclose to
> > others
> > > this message or any attachment. Please also notify the sender by
> replying
> > > to this email or by telephone (+44(020 7896 0011) and then delete the
> > email
> > > and any copies of it. Opinions, conclusion (etc) that do not relate to
> > the
> > > official business of this company shall be understood as neither given
> > nor
> > > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > > registered in England and Wales, company number 04008957) and IG Index
> > > Limited (a company registered in England and Wales, company number
> > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> IG
> > > Index Limited (register number 114059) are authorised and regulated by
> > the
> > > Financial Conduct Authority.
> > >
> >
>

[jira] [Commented] (KAFKA-2442) QuotasTest should not fail when cpu is busy

[Dong Lin (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15767765#comment-15767765
 ] 

Dong Lin commented on KAFKA-2442:
-

[~guozhang] I lost track of this issue and I am not sure if it is still valid. 
Given that this only affect tests and there is no ongoing complain of this 
issue that I am aware of, I think we close this ticket now.

> QuotasTest should not fail when cpu is busy
> ---
>
> Key: KAFKA-2442
> URL: https://issues.apache.org/jira/browse/KAFKA-2442
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Dong Lin
>Assignee: Aditya Auradkar
>  Labels: transient-unit-test-failure
>
> We observed that testThrottledProducerConsumer in QuotasTest may fail or 
> succeed randomly. It appears that the test may fail when the system is slow. 
> We can add timer in the integration test to avoid random failure.
> See an example failure at 
> https://builds.apache.org/job/kafka-trunk-git-pr/166/console for patch 
> https://github.com/apache/kafka/pull/142.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-2442) QuotasTest should not fail when cpu is busy

[Dong Lin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dong Lin resolved KAFKA-2442.
-
Resolution: Won't Fix

> QuotasTest should not fail when cpu is busy
> ---
>
> Key: KAFKA-2442
> URL: https://issues.apache.org/jira/browse/KAFKA-2442
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Dong Lin
>Assignee: Aditya Auradkar
>  Labels: transient-unit-test-failure
>
> We observed that testThrottledProducerConsumer in QuotasTest may fail or 
> succeed randomly. It appears that the test may fail when the system is slow. 
> We can add timer in the integration test to avoid random failure.
> See an example failure at 
> https://builds.apache.org/job/kafka-trunk-git-pr/166/console for patch 
> https://github.com/apache/kafka/pull/142.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] KIP-92 - Add per partition lag metrics to KafkaConsumer

[radai]

+1

On Wed, Dec 21, 2016 at 9:51 AM, Dong Lin  wrote:

> +1 (non-binding)
>
> On Thu, Dec 15, 2016 at 5:32 PM, Becket Qin  wrote:
>
> > Hi,
> >
> > I want to start a voting thread on KIP-92 which proposes to add per
> > partition lag metrics to KafkaConsumer. The KIP wiki page is below:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 92+-+Add+per+partition+lag+metrics+to+KafkaConsumer
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
>

Re: [DISCUSS] KIP-92 - Add per partition lag metrics to KafkaConsumer

[Joel Koshy]

LGTM. However, can you comment on the effect of releasing ownership of
partitions after a rebalance? For e.g., should it reset itself to (say) -1?
or removed? This really applies to any per-partition metrics that we intend
to maintain in the consumer.

On Mon, Nov 14, 2016 at 9:35 AM, Becket Qin  wrote:

> Hey Michael,
>
> Thanks for the comments. Exposing the lag on the client side may serve some
> cases a little different from monitoring. For example, one of the use case
> we have was that an application has some high priority and low priority
> topics to consume. They want to switch between consuming from high priority
> and low priority topics alternately based on the lag on the partition. i.e.
> if the consume has already caught up with the log end offset of a high
> priority topic, the application will switch to consume from the low
> priority topics. Otherwise it will continue consuming from the high
> priority topics. We have seen a few other similar use cases that require a
> programmatic access to the lag. Although people can always use
> offsetsForTimes() to get the LEO, but it is more expensive call involving
> an RPC and is a blocking call.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
>
>
> On Mon, Nov 14, 2016 at 9:18 AM, Michael Pearce 
> wrote:
>
> > Should state I have no objections adding this client side, just more a
> > question to why we don't look and propose to add this broker side also.
> >
> > Sent using OWA for iPhone
> > 
> > From: Michael Pearce 
> > Sent: Monday, November 14, 2016 4:58:45 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-92 - Add per partition lag metrics to
> > KafkaConsumer
> >
> > Why do we not look to expose the lag broker side centrally?
> >
> > Eg like burrow.
> >
> > From an operations point it's a lot easier to monitor lag centrally than
> > per application. Also then you'd be able to see lag of consumers not
> alive
> > or stalled.
> >
> > The information if the consumer uses Kafka based or zookeeper offsets is
> > available to the broker.
> > 
> > From: Becket Qin 
> > Sent: Sunday, November 13, 2016 4:13:01 AM
> > To: dev@kafka.apache.org
> > Subject: [DISCUSS] KIP-92 - Add per partition lag metrics to
> KafkaConsumer
> >
> > Hi,
> >
> > We created KIP-92 to propose adding per partition lag metrics to
> > KafkaConsumer.
> >
> > The KIP wiki link is the following:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 92+-+Add+per+partition+lag+metrics+to+KafkaConsumer
> >
> > Comments are welcome.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> > The information contained in this email is strictly confidential and for
> > the use of the addressee only, unless otherwise indicated. If you are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by replying
> > to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> > and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> > official business of this company shall be understood as neither given
> nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> > Index Limited (register number 114059) are authorised and regulated by
> the
> > Financial Conduct Authority.
> >
>

[jira] [Commented] (KAFKA-2442) QuotasTest should not fail when cpu is busy

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15767720#comment-15767720
 ] 

Guozhang Wang commented on KAFKA-2442:
--

[~lindong] Is this still a valid issue?

> QuotasTest should not fail when cpu is busy
> ---
>
> Key: KAFKA-2442
> URL: https://issues.apache.org/jira/browse/KAFKA-2442
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Dong Lin
>Assignee: Aditya Auradkar
>  Labels: transient-unit-test-failure
>
> We observed that testThrottledProducerConsumer in QuotasTest may fail or 
> succeed randomly. It appears that the test may fail when the system is slow. 
> We can add timer in the integration test to avoid random failure.
> See an example failure at 
> https://builds.apache.org/job/kafka-trunk-git-pr/166/console for patch 
> https://github.com/apache/kafka/pull/142.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2170) 10 LogTest cases failed for file.renameTo failed under windows

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15767712#comment-15767712
 ] 

Guozhang Wang commented on KAFKA-2170:
--

Is this PR still valid in tackling this issue? 
https://github.com/apache/kafka/pull/154

> 10 LogTest cases failed for  file.renameTo failed under windows
> ---
>
> Key: KAFKA-2170
> URL: https://issues.apache.org/jira/browse/KAFKA-2170
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.10.1.0
> Environment: Windows
>Reporter: Honghai Chen
>Assignee: Jay Kreps
>
> get latest code from trunk, then run test 
> gradlew  -i core:test --tests kafka.log.LogTest
> Got 10 cases failed for same reason:
> kafka.common.KafkaStorageException: Failed to change the log file suffix from 
>  to .deleted for log segment 0
>   at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:259)
>   at kafka.log.Log.kafka$log$Log$$asyncDeleteSegment(Log.scala:756)
>   at kafka.log.Log.kafka$log$Log$$deleteSegment(Log.scala:747)
>   at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:514)
>   at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:514)
>   at scala.collection.immutable.List.foreach(List.scala:318)
>   at kafka.log.Log.deleteOldSegments(Log.scala:514)
>   at kafka.log.LogTest.testAsyncDelete(LogTest.scala:633)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:601)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:44)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:180)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:41)
>   at org.junit.runners.ParentRunner$1.evaluate(ParentRunner.java:173)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:220)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:86)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:49)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:69)
>   at 
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:48)
>   at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:601)
>   at 
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
>   at 
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
>   at 
> org.gradle.messaging.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
>   at 
> org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
>   at $Proxy2.processTestClass(Unknown Source)
>   at 
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:105)
>   at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:601)
>   at 
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
>   at 
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
>   at 
> org.gradle.messaging.remote.internal.hub.

[jira] [Commented] (KAFKA-1194) The kafka broker cannot delete the old log files after the configured time

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15767709#comment-15767709
 ] 

Guozhang Wang commented on KAFKA-1194:
--

Is this PR trying to tackle this issue at all? 
https://github.com/apache/kafka/pull/154

> The kafka broker cannot delete the old log files after the configured time
> --
>
> Key: KAFKA-1194
> URL: https://issues.apache.org/jira/browse/KAFKA-1194
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.8.1
> Environment: window
>Reporter: Tao Qin
>Assignee: Jay Kreps
>  Labels: features, patch
> Fix For: 0.10.2.0
>
> Attachments: KAFKA-1194.patch, Untitled.jpg, kafka-1194-v1.patch, 
> kafka-1194-v2.patch, screenshot-1.png
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> We tested it in windows environment, and set the log.retention.hours to 24 
> hours.
> # The minimum age of a log file to be eligible for deletion
> log.retention.hours=24
> After several days, the kafka broker still cannot delete the old log file. 
> And we get the following exceptions:
> [2013-12-19 01:57:38,528] ERROR Uncaught exception in scheduled task 
> 'kafka-log-retention' (kafka.utils.KafkaScheduler)
> kafka.common.KafkaStorageException: Failed to change the log file suffix from 
>  to .deleted for log segment 1516723
>  at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:249)
>  at kafka.log.Log.kafka$log$Log$$asyncDeleteSegment(Log.scala:638)
>  at kafka.log.Log.kafka$log$Log$$deleteSegment(Log.scala:629)
>  at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:418)
>  at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:418)
>  at 
> scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
>  at scala.collection.immutable.List.foreach(List.scala:76)
>  at kafka.log.Log.deleteOldSegments(Log.scala:418)
>  at 
> kafka.log.LogManager.kafka$log$LogManager$$cleanupExpiredSegments(LogManager.scala:284)
>  at 
> kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:316)
>  at 
> kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:314)
>  at 
> scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:743)
>  at scala.collection.Iterator$class.foreach(Iterator.scala:772)
>  at 
> scala.collection.JavaConversions$JIteratorWrapper.foreach(JavaConversions.scala:573)
>  at scala.collection.IterableLike$class.foreach(IterableLike.scala:73)
>  at 
> scala.collection.JavaConversions$JListWrapper.foreach(JavaConversions.scala:615)
>  at 
> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:742)
>  at kafka.log.LogManager.cleanupLogs(LogManager.scala:314)
>  at 
> kafka.log.LogManager$$anonfun$startup$1.apply$mcV$sp(LogManager.scala:143)
>  at kafka.utils.KafkaScheduler$$anon$1.run(KafkaScheduler.scala:100)
>  at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>  at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>  at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>  at java.lang.Thread.run(Thread.java:724)
> I think this error happens because kafka tries to rename the log file when it 
> is still opened.  So we should close the file first before rename.
> The index file uses a special data structure, the MappedByteBuffer. Javadoc 
> describes it as:
> A mapped byte buffer and the file mapping that it represents remain valid 
> until the buffer itself is garbage-collected.
> Fortunately, I find a forceUnmap function in kafka code, and perhaps it can 
> be used to free the MappedByteBuffer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2358) Cluster collection returning methods should never return null

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-2358:
-
Assignee: Stevo Slavic

> Cluster collection returning methods should never return null
> -
>
> Key: KAFKA-2358
> URL: https://issues.apache.org/jira/browse/KAFKA-2358
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.9.0.0
>Reporter: Stevo Slavic
>Assignee: Stevo Slavic
>Priority: Minor
>
> {{KafkaConsumer.partitionsFor}} method by it's signature returns a 
> {{List}}. Problem is that in case (metadata for) topic does 
> not exist, current implementation will return null, which is considered a bad 
> practice - instead of null it should return empty list.
> Root cause is that the Cluster collection returning methods are returning 
> null.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2341) Need Standard Deviation Metrics in MetricsBench

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15767672#comment-15767672
 ] 

Guozhang Wang commented on KAFKA-2341:
--

[~sebadiaz] Added you to the contributor list.

> Need Standard Deviation Metrics in MetricsBench
> ---
>
> Key: KAFKA-2341
> URL: https://issues.apache.org/jira/browse/KAFKA-2341
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 0.8.2.1
>Reporter: sebastien diaz
>Assignee: sebastien diaz
>Priority: Minor
>
> The standard deviation is a measure that is used to quantify the amount of 
> variation or dispersion of a set of data values.
> Very useful. Could be added to other sensors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2341) Need Standard Deviation Metrics in MetricsBench

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-2341:
-
Assignee: sebastien diaz

> Need Standard Deviation Metrics in MetricsBench
> ---
>
> Key: KAFKA-2341
> URL: https://issues.apache.org/jira/browse/KAFKA-2341
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 0.8.2.1
>Reporter: sebastien diaz
>Assignee: sebastien diaz
>Priority: Minor
>
> The standard deviation is a measure that is used to quantify the amount of 
> variation or dispersion of a set of data values.
> Very useful. Could be added to other sensors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] KIP-92 - Add per partition lag metrics to KafkaConsumer

[Dong Lin]

+1 (non-binding)

On Thu, Dec 15, 2016 at 5:32 PM, Becket Qin  wrote:

> Hi,
>
> I want to start a voting thread on KIP-92 which proposes to add per
> partition lag metrics to KafkaConsumer. The KIP wiki page is below:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 92+-+Add+per+partition+lag+metrics+to+KafkaConsumer
>
> Thanks,
>
> Jiangjie (Becket) Qin
>

[GitHub] kafka pull request #2284: MINOR: KStream JavaDoc fix

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2284


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-4533) subscribe() then poll() on new topic is very slow when subscribed to many topics

[Sergey Alaev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Alaev updated KAFKA-4533:

Attachment: sgs.log.tar.gz

org.apache.kafka.* TRACE logs

> subscribe() then poll() on new topic is very slow when subscribed to many 
> topics
> 
>
> Key: KAFKA-4533
> URL: https://issues.apache.org/jira/browse/KAFKA-4533
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.10.1.0
>Reporter: Sergey Alaev
> Attachments: sgs.log.tar.gz
>
>
> Given following case:
> consumer.subscribe(my_new_topic, [249 existing topics])
> publisher.send(my_new_topic, key, value)
> poll(10) until data from my_new_topic arrives
> I see data from `my_new_topic` only after approx. 90 seconds.
> If I subscribe only to my_new_topic, I get results within seconds.
> Logs contain lots of lines like this:
> 19:28:07.972 [kafka-thread] DEBUG 
> org.apache.kafka.clients.consumer.internals.Fetcher - Resetting offset for 
> partition demo.com_recipient-2-0 to earliest offset.
> 19:28:08.247 [kafka-thread] DEBUG 
> org.apache.kafka.clients.consumer.internals.Fetcher - Fetched {timestamp=-1, 
> offset=0} for partition demo.com_recipient-2-0
> Probably you should do that in batch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] 0.10.1.1 RC1

[Ismael Juma]

Hi Guozhang,

Thanks for driving this release as it includes a few important fixes. I
belatedly tested Scala 2.12 binaries using the quickstart,  +1
(non-binding) from me.

Ismael

On Thu, Dec 15, 2016 at 1:29 PM, Guozhang Wang  wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the second, and hopefully the last candidate for the release of
> Apache Kafka 0.10.1.1 before the break. This is a bug fix release and it
> includes fixes and improvements from 30 JIRAs. See the release notes for
> more details:
>
> http://home.apache.org/~guozhang/kafka-0.10.1.1-rc1/RELEASE_NOTES.html
>
> *** Please download, test and vote by Tuesday, 20 December, 8pm PT ***
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> http://home.apache.org/~guozhang/kafka-0.10.1.1-rc1/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> NOTE the artifacts include the ones built from Scala 2.12.1 and Java8,
> which are treated a pre-alpha artifacts for the Scala community to try and
> test it out:
>
> https://repository.apache.org/content/groups/staging/org/apa
> che/kafka/kafka_2.12/0.10.1.1/
>
> We will formally add the scala 2.12 support in future minor releases.
>
>
> * Javadoc:
> http://home.apache.org/~guozhang/kafka-0.10.1.1-rc1/javadoc/
>
> * Tag to be voted upon (off 0.10.0 branch) is the 0.10.0.1-rc0 tag:
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=
> c3638376708ee6c02dfe4e57747acae0126fa6e7
>
>
> Thanks,
> Guozhang
>
> --
> -- Guozhang
>

[jira] [Updated] (KAFKA-1573) Transient test failures on LogTest.testCorruptLog

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-1573:
---
Fix Version/s: (was: 0.10.1.1)

> Transient test failures on LogTest.testCorruptLog
> -
>
> Key: KAFKA-1573
> URL: https://issues.apache.org/jira/browse/KAFKA-1573
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Guozhang Wang
>Assignee: Priyank Shah
>  Labels: transient-unit-test-failure
>
> Here is an example of the test failure trace:
> junit.framework.AssertionFailedError: expected:<87> but was:<68>
>   at junit.framework.Assert.fail(Assert.java:47)
>   at junit.framework.Assert.failNotEquals(Assert.java:277)
>   at junit.framework.Assert.assertEquals(Assert.java:64)
>   at junit.framework.Assert.assertEquals(Assert.java:130)
>   at junit.framework.Assert.assertEquals(Assert.java:136)
>   at 
> kafka.log.LogTest$$anonfun$testCorruptLog$1.apply$mcVI$sp(LogTest.scala:615)
>   at 
> scala.collection.immutable.Range$ByOne$class.foreach$mVc$sp(Range.scala:282)
>   at 
> scala.collection.immutable.Range$$anon$2.foreach$mVc$sp(Range.scala:265)
>   at kafka.log.LogTest.testCorruptLog(LogTest.scala:595)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at 
> org.junit.internal.runners.TestMethodRunner.executeMethodBody(TestMethodRunner.java:99)
>   at 
> org.junit.internal.runners.TestMethodRunner.runUnprotected(TestMethodRunner.java:81)
>   at 
> org.junit.internal.runners.BeforeAndAfterRunner.runProtected(BeforeAndAfterRunner.java:34)
>   at 
> org.junit.internal.runners.TestMethodRunner.runMethod(TestMethodRunner.java:75)
>   at 
> org.junit.internal.runners.TestMethodRunner.run(TestMethodRunner.java:45)
>   at 
> org.junit.internal.runners.TestClassMethodsRunner.invokeTestMethod(TestClassMethodsRunner.java:71)
>   at 
> org.junit.internal.runners.TestClassMethodsRunner.run(TestClassMethodsRunner.java:35)
>   at 
> org.junit.internal.runners.TestClassRunner$1.runUnprotected(TestClassRunner.java:42)
>   at 
> org.junit.internal.runners.BeforeAndAfterRunner.runProtected(BeforeAndAfterRunner.java:34)
>   at 
> org.junit.internal.runners.TestClassRunner.run(TestClassRunner.java:52)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:80)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:47)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:69)
>   at 
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:49)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at 
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
>   at 
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
>   at 
> org.gradle.messaging.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
>   at 
> org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
>   at $Proxy2.processTestClass(Unknown Source)
>   at 
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:103)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at 
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
>   at 
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
>   at 
> org.gradle.messaging.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:355)
>   at 
> org.gradle.internal.concurrent.DefaultExecutorFactory$StoppableExecutorImpl$1.run(DefaultExecutorFactory.java:66)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>   at 
>

[jira] [Updated] (KAFKA-4186) Transient failure in KStreamAggregationIntegrationTest.shouldGroupByKey

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-4186:
---
Fix Version/s: (was: 0.10.1.1)

> Transient failure in KStreamAggregationIntegrationTest.shouldGroupByKey
> ---
>
> Key: KAFKA-4186
> URL: https://issues.apache.org/jira/browse/KAFKA-4186
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Reporter: Jason Gustafson
>Assignee: Damian Guy
>
> Saw this running locally off of trunk:
> {code}
> org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest > 
> shouldGroupByKey[1] FAILED
> java.lang.AssertionError: Condition not met within timeout 6. Did not 
> receive 10 number of records
> at 
> org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:268)
> at 
> org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived(IntegrationTestUtils.java:211)
> at 
> org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest.receiveMessages(KStreamAggregationIntegrationTest.java:480)
> at 
> org.apache.kafka.streams.integration.KStreamAggregationIntegrationTest.shouldGroupByKey(KStreamAggregationIntegrationTest.java:407)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-4565) Separate Internal and External traffic

[Ismael Juma (JIRA)]

Ismael Juma created KAFKA-4565:
--

 Summary: Separate Internal and External traffic
 Key: KAFKA-4565
 URL: https://issues.apache.org/jira/browse/KAFKA-4565
 Project: Kafka
  Issue Type: Bug
Reporter: Ismael Juma
Assignee: Ismael Juma


During the 0.9.0.0 release cycle, support for multiple listeners per broker was 
introduced (KAFKA-1809). Each listener is associated with a security protocol, 
ip/host and port. When combined with the advertised listeners mechanism, there 
is a fair amount of flexibility with one limitation: at most one listener per 
security protocol in each of the two configs (listeners and 
advertised.listeners).

In some environments, one may want to differentiate between external clients, 
internal clients and replication traffic independently of the security protocol 
for cost, performance and security reasons. See the KIP for more details: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-103%3A+Separate+Internal+and+External+traffic



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2223: Update `createFile` function of Uitls.scala

[Allianzcortex]

Github user Allianzcortex closed the pull request at:

https://github.com/apache/kafka/pull/2223


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-4229) Controller can't start after several zk expired event

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-4229:
---
Fix Version/s: 0.10.2.0

> Controller can't start after several zk expired event
> -
>
> Key: KAFKA-4229
> URL: https://issues.apache.org/jira/browse/KAFKA-4229
> Project: Kafka
>  Issue Type: Bug
>  Components: controller
>Affects Versions: 0.9.0.0, 0.9.0.1, 0.10.0.0, 0.10.0.1
>Reporter: Pengwei
>Assignee: Pengwei
>  Labels: reliability
> Fix For: 0.10.2.0
>
>
> We found the controller not started after several zk expired event in our 
> test environment.  By analysing the log, I found the controller will handle 
> the ephemeral node data delete event first and then the zk expired event , 
> then the controller will gone.
> I can reproducer it on my develop env:
> 1. set up a one broker and one zk env, specify a very large zk timeout (20s)
> 2. stop the broker and remove the zk's /broker/ids/0  directory
> 3. restart the broker and make  a breakpoint in the zk client's event thread 
> to queue the delete event.
> 4. after the /controller node gone the breakpoint will hit.
> 5. expired the current session(suspend the send thread) and create a new 
> session s2
> 6. resume the event thread, then the controller will handle 
> LeaderChangeListener.handleDataDeleted  and become leader
> 7. then controller will handle SessionExpirationListener.handleNewSession, it 
> resign the controller and elect,  but when elect it found the /controller 
> node is exist and not become the leader.  But the /controller node is created 
> by current session s2 will not remove. So the controller is gone



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Reg: ACLS

[Ismael Juma]

Yes, that's correct.

Ismael

On Sat, Dec 10, 2016 at 11:02 AM, BigData dev 
wrote:

> Hi Ashish, Ismael
> Thanks for Info.
> So on Kafka Cluster (With out any security enabled) I can add ACLS with IP
> address.
> Is that correct?
>
>
> Thanks,
> Bharat
>
>
> On Fri, Dec 9, 2016 at 11:14 AM, Ashish Singh  wrote:
>
> > Ismael, thanks for the correction. I assumed the question was targeted
> for
> > without any security enabled, but yea even then IP based auth is
> possible.
> >
> > On Fri, Dec 9, 2016 at 11:01 AM, Ismael Juma  wrote:
> >
> > > It is possible to use ACLs with IPs or other SASL mechanisms (PLAIN for
> > > example). So Kerberos and SSL are not required (although commonly
> used).
> > >
> > > Ismael
> > >
> > > On Fri, Dec 9, 2016 at 6:59 PM, Ashish Singh 
> > wrote:
> > >
> > > > Hey,
> > > >
> > > > No it does not. Without kerberos or ssl, all requests will appear to
> > come
> > > > from anonymous user, and as long as a user is not identified it is
> not
> > > > possible to do authorization on.
> > > >
> > > > On Fri, Dec 9, 2016 at 10:40 AM, BigData dev <
> bigdatadev...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi All,
> > > > > I have a question here, Does Kafka support ACL's with out
> > kerberos/SSL?
> > > > >
> > > > > Any info on this would be greatly helpful.
> > > > >
> > > > >
> > > > > Thanks
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Regards,
> > > > Ashish
> > > >
> > >
> >
> >
> >
> > --
> >
> > Regards,
> > Ashish
> >
>

[jira] [Commented] (KAFKA-4533) subscribe() then poll() on new topic is very slow when subscribed to many topics

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15767243#comment-15767243
 ] 

Ismael Juma commented on KAFKA-4533:


Thanks for the report. The requests are done in batches. We'd need more 
information to understand the issue. Maybe you can attach more of the log?

> subscribe() then poll() on new topic is very slow when subscribed to many 
> topics
> 
>
> Key: KAFKA-4533
> URL: https://issues.apache.org/jira/browse/KAFKA-4533
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.10.1.0
>Reporter: Sergey Alaev
>
> Given following case:
> consumer.subscribe(my_new_topic, [249 existing topics])
> publisher.send(my_new_topic, key, value)
> poll(10) until data from my_new_topic arrives
> I see data from `my_new_topic` only after approx. 90 seconds.
> If I subscribe only to my_new_topic, I get results within seconds.
> Logs contain lots of lines like this:
> 19:28:07.972 [kafka-thread] DEBUG 
> org.apache.kafka.clients.consumer.internals.Fetcher - Resetting offset for 
> partition demo.com_recipient-2-0 to earliest offset.
> 19:28:08.247 [kafka-thread] DEBUG 
> org.apache.kafka.clients.consumer.internals.Fetcher - Fetched {timestamp=-1, 
> offset=0} for partition demo.com_recipient-2-0
> Probably you should do that in batch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4547) Consumer.position returns incorrect results for Kafka 0.10.1.0 client

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15767226#comment-15767226
 ] 

Ismael Juma commented on KAFKA-4547:


It would be good to fix this for 0.10.2.0, assuming it's not intentional. If it 
is intentional, we need to explain why and document it clearly.

> Consumer.position returns incorrect results for Kafka 0.10.1.0 client
> -
>
> Key: KAFKA-4547
> URL: https://issues.apache.org/jira/browse/KAFKA-4547
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.10.1.0
> Environment: Windows Kafka 0.10.1.0
>Reporter: Pranav Nakhe
>Assignee: Vahid Hashemian
>  Labels: clients
> Fix For: 0.10.2.0
>
>
> Consider the following code -
>   KafkaConsumer consumer = new 
> KafkaConsumer(props);
>   List listOfPartitions = new ArrayList();
>   for (int i = 0; i < 
> consumer.partitionsFor("IssueTopic").size(); i++) {
>   listOfPartitions.add(new TopicPartition("IssueTopic", 
> i));
>   }
>   consumer.assign(listOfPartitions);  
>   consumer.pause(listOfPartitions);
>   consumer.seekToEnd(listOfPartitions);
> //consumer.resume(listOfPartitions); -- commented out
>   for(int i = 0; i < listOfPartitions.size(); i++) {
>   
> System.out.println(consumer.position(listOfPartitions.get(i)));
>   }
>   
> I have created a topic IssueTopic with 3 partitions with a single replica on 
> my single node kafka installation (0.10.1.0)
> The behavior noticed for Kafka client 0.10.1.0 as against Kafka client 
> 0.10.0.1
> A) Initially when there are no messages on IssueTopic running the above 
> program returns
> 0.10.1.0   
> 0  
> 0  
> 0   
> 0.10.0.1
> 0
> 0
> 0
> B) Next I send 6 messages and see that the messages have been evenly 
> distributed across the three partitions. Running the above program now 
> returns 
> 0.10.1.0   
> 0  
> 0  
> 2  
> 0.10.0.1
> 2
> 2
> 2
> Clearly there is a difference in behavior for the 2 clients.
> Now after seekToEnd call if I make a call to resume (uncomment the resume 
> call in code above) then the behavior is
> 0.10.1.0   
> 2  
> 2  
> 2  
> 0.10.0.1
> 2
> 2
> 2
> This is an issue I came across when using the spark kafka integration for 
> 0.10. When I use kafka 0.10.1.0 I started seeing this issue. I had raised a 
> pull request to resolve that issue [SPARK-18779] but when looking at the 
> kafka client implementation/documentation now it seems the issue is with 
> kafka and not with spark. There does not seem to be any documentation which 
> specifies/implies that we need to call resume after seekToEnd for position to 
> return the correct value. Also there is a clear difference in the behavior in 
> the two kafka client implementations. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4547) Consumer.position returns incorrect results for Kafka 0.10.1.0 client

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-4547:
---
Fix Version/s: 0.10.2.0

> Consumer.position returns incorrect results for Kafka 0.10.1.0 client
> -
>
> Key: KAFKA-4547
> URL: https://issues.apache.org/jira/browse/KAFKA-4547
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.10.1.0
> Environment: Windows Kafka 0.10.1.0
>Reporter: Pranav Nakhe
>Assignee: Vahid Hashemian
>  Labels: clients
> Fix For: 0.10.2.0
>
>
> Consider the following code -
>   KafkaConsumer consumer = new 
> KafkaConsumer(props);
>   List listOfPartitions = new ArrayList();
>   for (int i = 0; i < 
> consumer.partitionsFor("IssueTopic").size(); i++) {
>   listOfPartitions.add(new TopicPartition("IssueTopic", 
> i));
>   }
>   consumer.assign(listOfPartitions);  
>   consumer.pause(listOfPartitions);
>   consumer.seekToEnd(listOfPartitions);
> //consumer.resume(listOfPartitions); -- commented out
>   for(int i = 0; i < listOfPartitions.size(); i++) {
>   
> System.out.println(consumer.position(listOfPartitions.get(i)));
>   }
>   
> I have created a topic IssueTopic with 3 partitions with a single replica on 
> my single node kafka installation (0.10.1.0)
> The behavior noticed for Kafka client 0.10.1.0 as against Kafka client 
> 0.10.0.1
> A) Initially when there are no messages on IssueTopic running the above 
> program returns
> 0.10.1.0   
> 0  
> 0  
> 0   
> 0.10.0.1
> 0
> 0
> 0
> B) Next I send 6 messages and see that the messages have been evenly 
> distributed across the three partitions. Running the above program now 
> returns 
> 0.10.1.0   
> 0  
> 0  
> 2  
> 0.10.0.1
> 2
> 2
> 2
> Clearly there is a difference in behavior for the 2 clients.
> Now after seekToEnd call if I make a call to resume (uncomment the resume 
> call in code above) then the behavior is
> 0.10.1.0   
> 2  
> 2  
> 2  
> 0.10.0.1
> 2
> 2
> 2
> This is an issue I came across when using the spark kafka integration for 
> 0.10. When I use kafka 0.10.1.0 I started seeing this issue. I had raised a 
> pull request to resolve that issue [SPARK-18779] but when looking at the 
> kafka client implementation/documentation now it seems the issue is with 
> kafka and not with spark. There does not seem to be any documentation which 
> specifies/implies that we need to call resume after seekToEnd for position to 
> return the correct value. Also there is a clear difference in the behavior in 
> the two kafka client implementations. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4426) Add consumer.close(timeout, unit) for graceful close with timeout

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-4426:
---
Fix Version/s: 0.10.2.0

> Add consumer.close(timeout, unit) for graceful close with timeout
> -
>
> Key: KAFKA-4426
> URL: https://issues.apache.org/jira/browse/KAFKA-4426
> Project: Kafka
>  Issue Type: Improvement
>  Components: consumer
>Affects Versions: 0.10.1.0
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
> Fix For: 0.10.2.0
>
>
> KAFKA-3703 implements graceful close of consumers with a hard-coded timeout 
> of 5 seconds. For consistency with the producer, add a close method with 
> configurable timeout for Consumer.
> {quote}
> public void close(long timeout, TimeUnit unit);
> {quote}
> Since this is a public interface change, this change requires a KIP.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-101: Alter Replication Protocol to use Leader Generation rather than High Watermark for Truncation

[Ben Stopford]

Thanks Apurva - yes that's one of those funny english phrases which are
often read one way, but are really quite ambiguous. I have reworded.

Thanks for pointing this one out. Appreciated.

B

On Tue, 20 Dec 2016 at 23:49, Apurva Mehta  wrote:

Hi Ben,



Thanks for the KIP. It is very well written and explains the problem and

solution very nicely. I have one --very minor-- question. In the 'steps'

section, you write:



> 4.6 The follower starts fetching from the leader from its log end offset.



The use of 'its' is a bit ambiguous here. I presume that you mean that the

follower fetches from the log end offset of the follower (and not the

leader). Might be worth clarifying whose log end offset is referred to

here.



While the perceived ambiguity may be put down to my english skills, I still

feet it would be better to leave no room for doubt.



Thanks,

Apurva



On Sun, Dec 11, 2016 at 4:30 AM, Ben Stopford  wrote:



> Hi All

>

> Please find the below KIP which describes a proposed solution to a couple

> of issues that have been observed with the replication protocol.

>

> In short, the proposal replaces the use of the High Watermark, for

> follower log trunctation, with an alternate Generation Marker. This

> uniquely defines which leader messages were acknowledged by.

>

> https://cwiki.apache.org/confluence/display/KAFKA/KIP-

> 101+-+Alter+Replication+Protocol+to+use+Leader+

> Generation+rather+than+High+Watermark+for+Truncation <

> https://cwiki.apache.org/confluence/display/KAFKA/KIP-

> 101+-+Alter+Replication+Protocol+to+use+Leader+

> Generation+rather+than+High+Watermark+for+Truncation>

>

> All comments and suggestions greatly appreciated.

>

> Ben Stopford

> Confluent, http://www.confluent.io 

>

>

Re: [DISCUSS] KIP-102 - Add close with timeout for consumers

[Rajini Sivaram]

I have added some more detail to the "Proposed Changes" section. Also
created a preliminary PR for the JIRA (
https://github.com/apache/kafka/pull/2285).

I am using *request.timeout.ms * to bound
individual requests during close (the KIP does not address timeouts in any
other code path) to ensure that *close()* always completes within a bounded
time even when timeout is not specified. This is similar to the producer
where requests are aborted after *request.timeout.ms
. *The PR contains unit and integration tests
for all the close scenarios I could think of (but there could be more).


On Mon, Dec 19, 2016 at 10:32 PM, Guozhang Wang  wrote:

> +1 on this idea as well.
>
> Streams has also added a similar feature itself partly because consumer
> does not support it directly (other part of the reason is that like
> brokers, streams also have some exception handling logic which could lead
> to deadlock with careless System.exit). For consumer itself I think the
> trickiness lies in the prefetching calls as well as commit / HB requests
> cleanup with the timeout, and I agree with Ewen that it's better to be
> merged in the early release cycle than a last minute merge.
>
>
>
> Guozhang
>
> On Mon, Dec 19, 2016 at 4:18 AM, Rajini Sivaram 
> wrote:
>
> > Thank you for the reviews.
> >
> > @Becket @Ewen, Agree that making all blocking calls have a timeout will
> be
> > trickier and hence the scope of this KIP is limited to close().
> >
> > @Jay Yes, this should definitely go into release notes, will make sure it
> > is added. I will add some integration tests with broker failures for
> > testing the timeout, but they cannot completely eliminate the risk of a
> > hang. Over time, hopefully system tests will help catch most issues.
> >
> >
> > On Sat, Dec 17, 2016 at 1:15 AM, Jay Kreps  wrote:
> >
> > > I think this is great. Sounds like one implication is that existing
> code
> > > that called close() and hit the timeout would now hang indefinitely. We
> > saw
> > > this kind of thing a lot in automated testing scenarios where people
> > don't
> > > correctly sequence their shutdown of client and server. I think this is
> > > okay, but might be good to include in the release notes.
> > >
> > > -jay
> > >
> > > On Thu, Dec 15, 2016 at 5:32 AM, Rajini Sivaram 
> > > wrote:
> > >
> > > Hi all,
> > >
> > >
> > >
> > >
> > >
> > > I have just created KIP-102 to add a new close method for consumers
> with
> > a
> > >
> > >
> > > timeout parameter, making Consumer consistent with Producer:
> > >
> > >
> > >
> > >
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 102+-+Add+close+with+timeout+for+consumers
> > >
> > >
> > >
> > >
> > >
> > > Comments and suggestions are welcome.
> > >
> > >
> > >
> > >
> > >
> > > Thank you...
> > >
> > >
> > >
> > >
> > >
> > > Regards,
> > >
> > >
> > >
> > >
> > >
> > > Rajini
> > >
> >
> >
> >
> > --
> > Regards,
> >
> > Rajini
> >
>
>
>
> --
> -- Guozhang
>

[jira] [Commented] (KAFKA-4426) Add consumer.close(timeout, unit) for graceful close with timeout

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15766945#comment-15766945
 ] 

ASF GitHub Bot commented on KAFKA-4426:
---

GitHub user rajinisivaram opened a pull request:

https://github.com/apache/kafka/pull/2285

KAFKA-4426: Add close with timeout for KafkaConsumer

Code corresponding to KIP-102.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rajinisivaram/kafka KAFKA-4426

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2285.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2285


commit 2945ad36d33cd6c586f4260da2fa9b0324d31ed4
Author: Rajini Sivaram 
Date:   2016-11-22T14:19:46Z

KAFKA-4426: Add close with timeout for KafkaConsumer




> Add consumer.close(timeout, unit) for graceful close with timeout
> -
>
> Key: KAFKA-4426
> URL: https://issues.apache.org/jira/browse/KAFKA-4426
> Project: Kafka
>  Issue Type: Improvement
>  Components: consumer
>Affects Versions: 0.10.1.0
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
>
> KAFKA-3703 implements graceful close of consumers with a hard-coded timeout 
> of 5 seconds. For consistency with the producer, add a close method with 
> configurable timeout for Consumer.
> {quote}
> public void close(long timeout, TimeUnit unit);
> {quote}
> Since this is a public interface change, this change requires a KIP.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2285: KAFKA-4426: Add close with timeout for KafkaConsum...

[rajinisivaram]

GitHub user rajinisivaram opened a pull request:

https://github.com/apache/kafka/pull/2285

KAFKA-4426: Add close with timeout for KafkaConsumer

Code corresponding to KIP-102.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rajinisivaram/kafka KAFKA-4426

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2285.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2285


commit 2945ad36d33cd6c586f4260da2fa9b0324d31ed4
Author: Rajini Sivaram 
Date:   2016-11-22T14:19:46Z

KAFKA-4426: Add close with timeout for KafkaConsumer




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [VOTE] KIP-84: Support SASL SCRAM mechanisms

[Rajini Sivaram]

Jun,

The KIP currently proposes to add 4 SASL mechanisms SCRAM-SHA-224,
SCRAM-SHA-256, SCRAM-SHA-384 and SCRAM-SHA-512. Ismael suggested supporting
just SCRAM-SHA-256 and SCRAM-SHA-512 to make it easier for non-Java client
support. What do you think?

Thank you,

Rajini


On Fri, Dec 2, 2016 at 2:44 PM, Ismael Juma  wrote:

> Thanks Rajini. Let's see what Jun says about limiting the number of SHA
> variants. Either way, +1 from me.
>
> Ismael
>
> On Fri, Dec 2, 2016 at 2:40 PM, Rajini Sivaram <
> rajinisiva...@googlemail.com
> > wrote:
>
> > Ismael,
> >
> > 1. Jun had suggested added the full list of SHA-nnn in the [DISCUSS]
> > thread. I am ok with limiting to a smaller number if required.
> >
> > 3. Added a section on security considerations to the KIP.
> >
> > Thank you,
> >
> > Rajini
> >
> > On Thu, Dec 1, 2016 at 4:22 PM, Ismael Juma  wrote:
> >
> > > Hi Rajini,
> > >
> > > Sorry for the delay. For some reason, both of your replies (for this
> and
> > > KIP-85) were marked as spam by Gmail. Comments inline.
> > >
> > > On Mon, Nov 28, 2016 at 3:47 PM, Rajini Sivaram <
> > > rajinisiva...@googlemail.com> wrote:
> > > >
> > > > 1. I think you had asked earlier for SCRAM-SHA-1 to be removed since
> it
> > > is
> > > > not secure :-) I am happy to add that back in so that clients which
> > don't
> > > > have access to a more secure algorithm can use it. But it would be a
> > > shame
> > > > to prevent users who only need Java clients from using more secure
> > > > mechanisms. Since SHA-1 is not secure, you need a secure Zookeeper
> > > > installation (or store your credentials in an alternative secure
> > store)..
> > > > By supporting multiple algorithms, we are giving the choice to users.
> > It
> > > > doesn't add much additional code, just the additional tests (one
> > > > integration test per mechanism). As more clients support new
> > mechanisms,
> > > > users can enable these without any changes to Kafka.
> > > >
> > >
> > > Yes, I remember that I asked for SCRAM-SHA-1 to be removed. I probably
> > > wasn't clear. My suggestion was not to add that back, but whether we
> > needed
> > > so many variants. For example, we could support SCRAM-SHA-256 and
> > > SCRAM-SHA-512.
> > > Would that be sufficient? It's true that the cost is not that large for
> > us,
> > > but every other client also has to pay that additional extra cost and I
> > am
> > > not sure sure about the benefit of some of the options.
> > >
> > > 3. I am assuming that ZK authentication will be enabled and ZK
> > > > configuration will be done directly using ZK commands. This is true
> for
> > > > ACLs, quotas etc. as well?
> > > >
> > >
> > > Right, I also thought that ACLs was the closest example. However, it
> > seems
> > > like getting read access to the SCRAM DB has potentially worse
> > > consequences:
> > >
> > > "For a specific secret compromised, if an exchange is obtained from the
> > > wire by some mechanism, this gives sufficient information for an
> imposter
> > > to pose as the client for that server (but not another one using the
> same
> > > password). Note that this interception is only useful if the database
> has
> > > been compromised – SCRAM is safe against replay attack. This is the
> > primary
> > > SCRAM weakness, and why it is important to protect the secret database
> > > carefully and to use TLS."[1]
> > >
> > > Also, because we are using fast hashes (instead of slow ones like
> bcrypt,
> > > scrypt, etc.), we are more susceptible to dictionary attacks
> (potentially
> > > mitigated by a reasonably large iteration count combined with good
> > quality
> > > passwords).
> > >
> > > If nothing else, it may be worth mentioning some of this in the KIP
> > and/or
> > > documentation.
> > >
> > > Ismael
> > >
> > > [1] http://www.isode.com/whitepapers/scram.html
> > >
> >
> >
> >
> > --
> > Regards,
> >
> > Rajini
> >
>

[GitHub] kafka pull request #2284: MINOR: KStream JavaDoc fix

[mjsax]

GitHub user mjsax opened a pull request:

https://github.com/apache/kafka/pull/2284

MINOR: KStream JavaDoc fix



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mjsax/kafka javaDoc

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2284.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2284


commit 581516d01ff858d0e4fbc487329df1b10b85a135
Author: Matthias J. Sax 
Date:   2016-12-21T10:16:56Z

MINOR: KStream JavaDoc fix




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] Control Messages - [Was: KIP-82 - Add Record Headers]

[Matthias J. Sax]

I agree with all. Just want to elaborate a few things:

3. There are two different use cases:
   (a) the one you describe -- I want to shutdown NOW and don't want to
wait -- I agree with your observations etc
   (b) we intentionally want to "drain" the stream processing topology
before shutting down -- yes, if I have lot of intermediate data this
might take some time, but I want/need a clean shutdown like this

Case 3(b) is currently not possible and exactly want we need for
"Incremental Batch KIP" -- there are other use case for 3(b), too.


4. The point about "it's just a client thing is true, but it should work
for client that are not aware of the messages, too. Ie, we need an
opt-in mechanism -- so some changes are required -- not to the brokers
though -- but it cannot be done "external" to the clients -- otherwise
people would need to change their client code.



About "embedded control message" vs "extra control message stream".
IMHO, there a use cases for both and both approaches complete each other
(they are not conflicting).


-Matthias



On 12/14/16 8:36 PM, Ignacio Solis wrote:
> I'm renaming this thread in case we start deep diving.
> 
> I'm in favor of so called "control messages", at least the notion of
> those.  However, I'm not sure about the design.
> 
> What I understood from the original mail:
> 
> A. Provide a message that does not get returned by poll()
> B. Provide a way for applications to consume these messages (sign up?)
> C. Control messages would be associated with a topic.
> D. Control messages should be _in_ the topic.
> 
> 
> 
> 1. The first thing to point out is that this can be done with headers.
> I assume that's why you sent it on the header thread. As you state, if
> we had headers, you would not require a separate KIP.  So, in a way,
> you're trying to provide a concrete use case for headers.  I wanted to
> separate the discussion to a separate thread mostly because while I
> like the idea, and I like the fact that it can be done by headers,
> people might want to discuss alternatives.
> 
> 2. I'm also assuming that you're intentionally trying to preserve
> order. Headers could do this natively of course. You could also
> achieve this with the separate topic given identifiers, sequence
> numbers, headers, etc.  However...
> 
> 3. There are a few use cases where ordering is important but
> out-of-band is even more important. We have a few large workloads
> where this is of interest to us.  Obviously we can achieve this with a
> separate topic, but having a control channel for a topic that can send
> high priority data would be interesting.   And yes, we would learn a
> lot form the TCP experiences with the urgent pointer (
> https://tools.ietf.org/html/rfc6093 ) and other out-of-band
> communication techniques.
> 
> You have an example of a "shutdown marker".  This works ok as a
> terminator, however, it is not very fast.  If I have 4 TB of data
> because of asynchronous processing, then a shutdown marker at the end
> of the 4TB is not as useful as having an out-of-band message that will
> tell me immediately that those 4TB should not be processed.   So, from
> this perspective, I prefer to have a separate topic and not embed
> control messages with the data.
> 
> If the messages are part of the data, or associated to specific data,
> then they should be in the data. If they are about process, we need an
> out-of-band mechanism.
> 
> 
> 4. The general feeling I have gotten from a few people on the list is:
> Why not just do this above the kafka clients?  After all, you could
> have a system to ignore certain schemas.
> 
> Effectively, if we had headers, it would be done from a client
> perspective, without the need to modify anything major.
> 
> If we wanted to do it with a separate topic, that could also be done
> without any broker changes. But you could imagine wanting some broker
> changes if the broker understands that 2 streams are tied together
> then it may make decisions based on that.  This would be similar to
> the handling of file system forks (
> https://en.wikipedia.org/wiki/Fork_(file_system) )
> 
> 
> 5. Also heard on discussions about headers: we don't know if this is
> generally useful. Maybe only a couple of institutions?  It may not be
> worth it to modify the whole stack for that.
> 
> I would again say that with headers you could pull it off easily, even
> if only for a subset of clients/applications wanted to use it.
> 
> 
> So, in summary. I like the idea.  I see benefits in implementing it
> through headers, but I also see benefits of having it as a separate
> stream.  I'm not too in favor of having a separate message handling
> pipeline for the same topic though.
> 
> Nacho
> 
> 
> 
> 
> 
> On Wed, Dec 14, 2016 at 9:51 AM, Matthias J. Sax  
> wrote:
>> Yes and no. I did overload the term "control message".
>>
>> EOS control messages are for client-broker communication and thus never
>> exposed to any application. And I think this is a good desig

Re: [VOTE] KIP-90 Remove zkClient dependency from Streams

[Matthias J. Sax]

+1


On 12/21/16 12:45 AM, Edoardo Comar wrote:
> +1 (non-binding) 
> thanks!
> --
> Edoardo Comar
> IBM MessageHub
> eco...@uk.ibm.com
> IBM UK Ltd, Hursley Park, SO21 2JN
> 
> IBM United Kingdom Limited Registered in England and Wales with number 
> 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 
> 3AU
> 
> 
> 
> From:   Ismael Juma 
> To: dev@kafka.apache.org
> Date:   20/12/2016 22:59
> Subject:Re: [VOTE] KIP-90 Remove zkClient dependency from Streams
> Sent by:isma...@gmail.com
> 
> 
> 
> Thanks for the KIP, +1 (binding).
> 
> On Tue, Dec 20, 2016 at 1:01 PM, Hojjat Jafarpour 
> wrote:
> 
>> Hi all,
>>
>> Seems that there is no opposition to this KIP. This email is to start 
> the
>> voting for this KIP.
>> Once again the KIP is for removing zkClient dependency from Streams. 
> Please
>> check out the KIP page:
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> 90+-+Remove+zkClient+
>> dependency+from+Streams
>>
>> Thanks,
>> --Hojjat
>>
> 
> 
> 
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number 
> 741598. 
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
> 



signature.asc
Description: OpenPGP digital signature

[jira] [Commented] (KAFKA-2331) Kafka does not spread partitions in a topic among all consumers evenly

[Jeff Widman (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15766509#comment-15766509
 ] 

Jeff Widman commented on KAFKA-2331:


Isn't this what round robin partitioning strategy was trying to solve?

If so, this issue should be closed.

> Kafka does not spread partitions in a topic among all consumers evenly
> --
>
> Key: KAFKA-2331
> URL: https://issues.apache.org/jira/browse/KAFKA-2331
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.1.1
>Reporter: Stefan Miklosovic
>
> I want to have 1 topic with 10 partitions. I am using default configuration 
> of Kafka. I create 1 topic with 10 partitions by that helper script and now I 
> am about to produce messages to it.
> The thing is that even all partitions are indeed consumed, some consumers 
> have more then 1 partition assigned even I have number of consumer threads 
> equal to partitions in a topic hence some threads are idle.
> Let's describe it in more detail.
> I know that common stuff that you need one consumer thread per partition. I 
> want to be able to commit offsets per partition and this is possible only 
> when I have 1 thread per consumer connector per partition (I am using high 
> level consumer).
> So I create 10 threads, in each thread I am calling 
> Consumer.createJavaConsumerConnector() where I am doing this
> topicCountMap.put("mytopic", 1);
> and in the end I have 1 iterator which consumes messages from 1 partition.
> When I do this 10 times, I have 10 consumers, consumer per thread per 
> partition where I can commit offsets independently per partition because if I 
> put different number from 1 in topic map, I would end up with more then 1 
> consumer thread for that topic for given consumer instance so if I am about 
> to commit offsets with created consumer instance, it would commit them for 
> all threads which is not desired.
> But the thing is that when I use consumers, only 7 consumers are involved and 
> it seems that other consumer threads are idle but I do not know why.
> The thing is that I am creating these consumer threads in a loop. So I start 
> first thread (submit to executor service), then another, then another and so 
> on.
> So the scenario is that first consumer gets all 10 partitions, then 2nd 
> connects so it is splits between these two to 5 and 5 (or something similar), 
> then other threads are connecting.
> I understand this as a partition rebalancing among all consumers so it 
> behaves well in such sense that if more consumers are being created, 
> partition rebalancing occurs between these consumers so every consumer should 
> have some partitions to operate upon.
> But from the results I see that there is only 7 consumers and according to 
> consumed messages it seems they are split like 3,2,1,1,1,1,1 partition-wise. 
> Yes, these 7 consumers covered all 10 partitions, but why consumers with more 
> then 1 partition do no split and give partitions to remaining 3 consumers?
> I am pretty much wondering what is happening with remaining 3 threads and why 
> they do not "grab" partitions from consumers which have more then 1 partition 
> assigned.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2019) RoundRobinAssignor clusters by consumer

[Jeff Widman (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15766498#comment-15766498
 ] 

Jeff Widman commented on KAFKA-2019:


As noted in KIP-54 
(https://cwiki.apache.org/confluence/display/pages/viewpage.action?pageId=62692483)
 this is not relevant to the new consumer.

Could someone update the issue title to make it clear it only applies to the 
old consumer?

I also suspect this may never get merged as the new consumer is the future, in 
which case it'd be nice if this were closed as "wontfix"


> RoundRobinAssignor clusters by consumer
> ---
>
> Key: KAFKA-2019
> URL: https://issues.apache.org/jira/browse/KAFKA-2019
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Reporter: Joseph Holsten
>Assignee: Neha Narkhede
>Priority: Minor
> Attachments: 0001-sort-consumer-thread-ids-by-hashcode.patch, 
> KAFKA-2019.patch
>
>
> When rolling out a change today, I noticed that some of my consumers are 
> "greedy", taking far more partitions than others.
> The cause is that the RoundRobinAssignor is using a list of ConsumerThreadIds 
> sorted by toString, which is {{ "%s-%d".format(consumer, threadId)}}. This 
> causes each consumer's threads to be adjacent to each other.
> One possible fix would be to define ConsumerThreadId.hashCode, and sort by 
> that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-3297) More optimally balanced partition assignment strategy (new consumer)

[Jeff Widman (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15766465#comment-15766465
 ] 

Jeff Widman edited comment on KAFKA-3297 at 12/21/16 8:38 AM:
--

Is this being superceded by KIP-54? 
https://cwiki.apache.org/confluence/display/pages/viewpage.action?pageId=62692483


was (Author: jeffwidman):
Was this KIP ever voted on? I see there's only a handful of messages about it, 
one of which mentions patching the round robin implementation to avoid 
"clumping" partitions from the same topic onto the same consumer. 

> More optimally balanced partition assignment strategy (new consumer)
> 
>
> Key: KAFKA-3297
> URL: https://issues.apache.org/jira/browse/KAFKA-3297
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Andrew Olson
>Assignee: Andrew Olson
> Fix For: 0.10.2.0
>
>
> While the roundrobin partition assignment strategy is an improvement over the 
> range strategy, when the consumer topic subscriptions are not identical 
> (previously disallowed but will be possible as of KAFKA-2172) it can produce 
> heavily skewed assignments. As suggested 
> [here|https://issues.apache.org/jira/browse/KAFKA-2172?focusedCommentId=14530767&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14530767]
>  it would be nice to have a strategy that attempts to assign an equal number 
> of partitions to each consumer in a group, regardless of how similar their 
> individual topic subscriptions are. We can accomplish this by tracking the 
> number of partitions assigned to each consumer, and having the partition 
> assignment loop assign each partition to a consumer interested in that topic 
> with the least number of partitions assigned. 
> Additionally, we can optimize the distribution fairness by adjusting the 
> partition assignment order:
> * Topics with fewer consumers are assigned first.
> * In the event of a tie for least consumers, the topic with more partitions 
> is assigned first.
> The general idea behind these two rules is to keep the most flexible 
> assignment choices available as long as possible by starting with the most 
> constrained partitions/consumers.
> This JIRA addresses the new consumer. For the original high-level consumer, 
> see KAFKA-2435.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3297) More optimally balanced partition assignment strategy (new consumer)

[Jeff Widman (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15766465#comment-15766465
 ] 

Jeff Widman commented on KAFKA-3297:


Was this KIP ever voted on? I see there's only a handful of messages about it, 
one of which mentions patching the round robin implementation to avoid 
"clumping" partitions from the same topic onto the same consumer. 

> More optimally balanced partition assignment strategy (new consumer)
> 
>
> Key: KAFKA-3297
> URL: https://issues.apache.org/jira/browse/KAFKA-3297
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Andrew Olson
>Assignee: Andrew Olson
> Fix For: 0.10.2.0
>
>
> While the roundrobin partition assignment strategy is an improvement over the 
> range strategy, when the consumer topic subscriptions are not identical 
> (previously disallowed but will be possible as of KAFKA-2172) it can produce 
> heavily skewed assignments. As suggested 
> [here|https://issues.apache.org/jira/browse/KAFKA-2172?focusedCommentId=14530767&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14530767]
>  it would be nice to have a strategy that attempts to assign an equal number 
> of partitions to each consumer in a group, regardless of how similar their 
> individual topic subscriptions are. We can accomplish this by tracking the 
> number of partitions assigned to each consumer, and having the partition 
> assignment loop assign each partition to a consumer interested in that topic 
> with the least number of partitions assigned. 
> Additionally, we can optimize the distribution fairness by adjusting the 
> partition assignment order:
> * Topics with fewer consumers are assigned first.
> * In the event of a tie for least consumers, the topic with more partitions 
> is assigned first.
> The general idea behind these two rules is to keep the most flexible 
> assignment choices available as long as possible by starting with the most 
> constrained partitions/consumers.
> This JIRA addresses the new consumer. For the original high-level consumer, 
> see KAFKA-2435.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2172) Round-robin partition assignment strategy too restrictive

[Jeff Widman (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15766445#comment-15766445
 ] 

Jeff Widman commented on KAFKA-2172:


Should this be marked as resolved?

KAFKA-2196 added this to the new consumer quite a while ago. 

KAFKA-2434 has a patch available but appears dead in the water, so if it's not 
getting merged then there's nothing more to do on this ticket.

> Round-robin partition assignment strategy too restrictive
> -
>
> Key: KAFKA-2172
> URL: https://issues.apache.org/jira/browse/KAFKA-2172
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Rosenberg
>
> The round-ropin partition assignment strategy, was introduced for the 
> high-level consumer, starting with 0.8.2.1.  This appears to be a very 
> attractive feature, but it has an unfortunate restriction, which prevents it 
> from being easily utilized.  That is that it requires all consumers in the 
> consumer group have identical topic regex selectors, and that they have the 
> same number of consumer threads.
> It turns out this is not always the case for our deployments.  It's not 
> unusual to run multiple consumers within a single process (with different 
> topic selectors), or we might have multiple processes dedicated for different 
> topic subsets.  Agreed, we could change these to have separate group ids for 
> each sub topic selector (but unfortunately, that's easier said than done).  
> In several cases, we do at least have separate client.ids set for each 
> sub-consumer, so it would be incrementally better if we could at least loosen 
> the requirement such that each set of topics selected by a groupid/clientid 
> pair are the same.
> But, if we want to do a rolling restart for a new version of a consumer 
> config, the cluster will likely be in a state where it's not possible to have 
> a single config until the full rolling restart completes across all nodes.  
> This results in a consumer outage while the rolling restart is happening.
> Finally, it's especially problematic if we want to canary a new version for a 
> period before rolling to the whole cluster.
> I'm not sure why this restriction should exist (as it obviously does not 
> exist for the 'range' assignment strategy).  It seems it could be made to 
> work reasonably well with heterogenous topic selection and heterogenous 
> thread counts.  The documentation states that "The round-robin partition 
> assignor lays out all the available partitions and all the available consumer 
> threads. It then proceeds to do a round-robin assignment from partition to 
> consumer thread."
> If the assignor can "lay out all the available partitions and all the 
> available consumer threads", it should be able to uniformly assign partitions 
> to the available threads.  In each case, if a thread belongs to a consumer 
> that doesn't have that partition selected, just move to the next available 
> thread that does have the selection, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2196) remove roundrobin identical topic constraint in consumer coordinator

[Jeff Widman (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15766440#comment-15766440
 ] 

Jeff Widman commented on KAFKA-2196:


What version did this land in?

> remove roundrobin identical topic constraint in consumer coordinator
> 
>
> Key: KAFKA-2196
> URL: https://issues.apache.org/jira/browse/KAFKA-2196
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Onur Karaman
>Assignee: Onur Karaman
> Attachments: KAFKA-2196.patch
>
>
> roundrobin doesn't need to make all consumers have identical topic 
> subscriptions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)