Message sent ordering guarantees

[郭旭]

Hi Kafka Experts,

(Sorry to send this question to DEV group, but it seems that I can not find
related document in user manual.)

For official document ,I can find message sent guarantee as below.
For *sync producer*, I think it is true but sync sent are very slow.(about
408 message per second if ack = all, 1000 message per second if ack = 1).

batch and async sent could satisfy our throughput requirement, but I'm not
sure if message sent ordering are guaranteed in *async *style.

For some critical application, for example( replicate mysql binlog to kafka
distributed committed log), binlog ordering are important(partitioned by
database/table/PK). throughput also important.

If I use async producer, partition the binlog by table and send them in
batch. Is it safe for binlog ordering for a single table?

Will async producer guarantee the send ordering?


Regards
Shawn

Guarantees At
a high-level Kafka gives the following guarantees:

   - Messages sent by a producer to a particular topic partition will be
   appended in the order they are sent. That is, if a message M1 is sent by
   the same producer as a message M2, and M1 is sent first, then M1 will have
   a lower offset than M2 and appear earlier in the log.

Re: Kafka consumers unable to process message

[Jason Gustafson]

The replicas are always trying to fetch new data from the partition leader.
When the leader fails, any in-flight fetches will fail and cause errors
such as the ones you saw in the broker log. Eventually the replicas will
discover the new leader and begin fetching again. And of course one of the
replicas will become the new leader.

-Jason

On Wed, Aug 31, 2016 at 8:04 PM, Ghosh, Achintya (Contractor) <
achintya_gh...@comcast.com> wrote:

> I'm trying get the consumer logs and will send you. So it means it can
> happen even my local datacenter too. Still I'm not understanding if 3 nodes
> are up and message already replicated why it's trying to fetch the data
> from failed node. Can you please explain bit details how it works. Thanks
> for your response.
>
> -Original Message-
> From: Jason Gustafson [mailto:ja...@confluent.io]
> Sent: Wednesday, August 31, 2016 10:56 PM
> To: us...@kafka.apache.org
> Cc: dev@kafka.apache.org
> Subject: Re: Kafka consumers unable to process message
>
> The exceptions show one of the replica fetcher threads on the broker
> failing which makes perfect sense since some of the partitions were bound
> to have leaders in the failed datacenter. I'd actually like to see the
> consumer logs at DEBUG level if possible.
>
> Thanks,
> Jason
>
> On Wed, Aug 31, 2016 at 7:48 PM, Ghosh, Achintya (Contractor) <
> achintya_gh...@comcast.com> wrote:
>
> > Hi Jason,
> >
> > No, I didn't bring down any zookeeper server. Even I tried with 3
> > zookeeper server one as an 'Observer' but the same issue.
> >
> > Here is the server log from one of the node of my other datacenter:
> >
> > [2016-09-01 01:25:19,221] INFO Truncating log TEST3-0 to offset 0.
> > (kafka.log.Log)
> > [2016-09-01 01:25:19,257] INFO [ReplicaFetcherThread-0-3], Starting
> > (kafka.server.ReplicaFetcherThread)
> > [2016-09-01 01:25:19,258] INFO [ReplicaFetcherManager on broker 4]
> > Added fetcher for partitions List([[TEST3,0], initOffset 0 to broker
> > BrokerEndPoint(3,psaq3-wc.sys.comcast.net,61616)] ) (kafka.server.
> > ReplicaFetcherManager)
> > [2016-09-01 01:26:14,154] WARN [ReplicaFetcherThread-0-3], Error in
> > fetch
> > kafka.server.ReplicaFetcherThread$FetchRequest@6618a925 (kafka.server.
> > ReplicaFetcherThread)
> > java.io.IOException: Connection to 3 was disconnected before the
> > response was read
> > at kafka.utils.NetworkClientBlockingOps$$anonfun$
> > blockingSendAndReceive$extension$1$$anonfun$apply$1.apply(
> > NetworkClientBlockingOps.scala:87)
> > at kafka.utils.NetworkClientBlockingOps$$anonfun$
> > blockingSendAndReceive$extension$1$$anonfun$apply$1.apply(
> > NetworkClientBlockingOps.scala:84)
> > at scala.Option.foreach(Option.scala:257)
> > at kafka.utils.NetworkClientBlockingOps$$anonfun$
> > blockingSendAndReceive$extension$1.apply(NetworkClientBlockingOps.
> > scala:84)
> > at kafka.utils.NetworkClientBlockingOps$$anonfun$
> > blockingSendAndReceive$extension$1.apply(NetworkClientBlockingOps.
> > scala:80)
> > at kafka.utils.NetworkClientBlockingOps$.recursivePoll$2(
> > NetworkClientBlockingOps.scala:137)
> > at kafka.utils.NetworkClientBlockingOps$.kafka$utils$
> > NetworkClientBlockingOps$$pollContinuously$extension(
> > NetworkClientBlockingOps.scala:143)
> > at
> > kafka.utils.NetworkClientBlockingOps$.blockingSendAndReceive$
> > extension(NetworkClientBlockingOps.scala:80)
> > at kafka.server.ReplicaFetcherThread.sendRequest(
> > ReplicaFetcherThread.scala:244)
> > at kafka.server.ReplicaFetcherThread.fetch(
> > ReplicaFetcherThread.scala:229)
> > at kafka.server.ReplicaFetcherThread.fetch(
> > ReplicaFetcherThread.scala:42)
> > at kafka.server.AbstractFetcherThread.processFetchRequest(
> > AbstractFetcherThread.scala:107)
> > at kafka.server.AbstractFetcherThread.doWork(
> > AbstractFetcherThread.scala:98)
> > at
> > kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
> > [2016-09-01 01:26:16,189] WARN [ReplicaFetcherThread-0-3], Error in
> > fetch
> > kafka.server.ReplicaFetcherThread$FetchRequest@6e7e2578 (kafka.server.
> > ReplicaFetcherThread)
> > java.io.IOException: Connection to psaq3-wc.sys.comcast.net:61616 (id:
> > 3
> > rack: null) failed
> > at
> > kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$
> > extension$2.apply(NetworkClientBlockingOps.scala:63)
> > at
> > kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$
> > extension$2.apply(NetworkClientBlockingOps.scala:59)
> > at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(
> > NetworkClientBlockingOps.scala:112)
> > at kafka.utils.NetworkClientBlockingOps$.kafka$utils$
> > NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.
> > scala:120)
> > at
> > kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(
> > NetworkClientBlockingOps.scala:59)
> > at 

[jira] [Commented] (KAFKA-4099) Change the time based log rolling to only based on the message timestamp.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15454198#comment-15454198
 ] 

ASF GitHub Bot commented on KAFKA-4099:
---

GitHub user becketqin opened a pull request:

https://github.com/apache/kafka/pull/1809

KAFKA-4099: Fix the potential frequent log rolling



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/becketqin/kafka KAFKA-4099

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1809.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1809


commit 06b9548feaa68c5f224d56df5e1d35bb0287fc8c
Author: Jiangjie Qin 
Date:   2016-09-01T03:24:52Z

KAFKA-4099: Fix the potential frequent log rolling.




> Change the time based log rolling to only based on the message timestamp.
> -
>
> Key: KAFKA-4099
> URL: https://issues.apache.org/jira/browse/KAFKA-4099
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.10.1.0
>
>
> This is an issue introduced in KAFKA-3163. When partition relocation occurs, 
> the newly created replica may have messages with old timestamp and cause the 
> log segment rolling for each message. The fix is to change the log rolling 
> behavior to only based on the message timestamp when the messages are in 
> message format 0.10.0 or above. If the first message in the segment does not 
> have a timetamp, we will fall back to use the wall clock time for log rolling.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1809: KAFKA-4099: Fix the potential frequent log rolling

[becketqin]

GitHub user becketqin opened a pull request:

https://github.com/apache/kafka/pull/1809

KAFKA-4099: Fix the potential frequent log rolling



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/becketqin/kafka KAFKA-4099

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1809.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1809


commit 06b9548feaa68c5f224d56df5e1d35bb0287fc8c
Author: Jiangjie Qin 
Date:   2016-09-01T03:24:52Z

KAFKA-4099: Fix the potential frequent log rolling.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-4099) Change the time based log rolling to only based on the message timestamp.

[Jiangjie Qin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jiangjie Qin updated KAFKA-4099:

Description: This is an issue introduced in KAFKA-3163. When partition 
relocation occurs, the newly created replica may have messages with old 
timestamp and cause the log segment rolling for each message. The fix is to 
change the log rolling behavior to only based on the message timestamp when the 
messages are in message format 0.10.0 or above. If the first message in the 
segment does not have a timetamp, we will fall back to use the wall clock time 
for log rolling.  (was: This is an issue introduced in KAFKA-3163. When 
partition relocation occurs, the newly created replica may have messages with 
old timestamp and cause the log segment rolling for each message. The fix is to 
change the log rolling behavior to only based on the message timestamp when the 
messages are in message format 0.10.0 or above.)

> Change the time based log rolling to only based on the message timestamp.
> -
>
> Key: KAFKA-4099
> URL: https://issues.apache.org/jira/browse/KAFKA-4099
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.10.1.0
>
>
> This is an issue introduced in KAFKA-3163. When partition relocation occurs, 
> the newly created replica may have messages with old timestamp and cause the 
> log segment rolling for each message. The fix is to change the log rolling 
> behavior to only based on the message timestamp when the messages are in 
> message format 0.10.0 or above. If the first message in the segment does not 
> have a timetamp, we will fall back to use the wall clock time for log rolling.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4099) Change the time based log rolling to only based on the message timestamp.

[Jiangjie Qin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jiangjie Qin updated KAFKA-4099:

Description: This is an issue introduced in KAFKA-3163. When partition 
relocation occurs, the newly created replica may have messages with old 
timestamp and cause the log segment rolling for each message. The fix is to 
change the log rolling behavior to only based on the message timestamp when the 
messages are in message format 0.10.0 or above.  (was: This is an issue 
introduced in KAFKA-3163. When partition relocation occurs, the newly created 
replica may have messages with old timestamp and cause the log segment rolling 
for each message. The fix is to change the log rolling behavior back to based 
on segment create time.)

> Change the time based log rolling to only based on the message timestamp.
> -
>
> Key: KAFKA-4099
> URL: https://issues.apache.org/jira/browse/KAFKA-4099
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.10.1.0
>
>
> This is an issue introduced in KAFKA-3163. When partition relocation occurs, 
> the newly created replica may have messages with old timestamp and cause the 
> log segment rolling for each message. The fix is to change the log rolling 
> behavior to only based on the message timestamp when the messages are in 
> message format 0.10.0 or above.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4099) Change the time based log rolling to only based on the message timestamp.

[Jiangjie Qin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jiangjie Qin updated KAFKA-4099:

Summary: Change the time based log rolling to only based on the message 
timestamp.  (was: Change the time based log rolling to base on the file create 
time instead of timestamp of the first message.)

> Change the time based log rolling to only based on the message timestamp.
> -
>
> Key: KAFKA-4099
> URL: https://issues.apache.org/jira/browse/KAFKA-4099
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.10.1.0
>
>
> This is an issue introduced in KAFKA-3163. When partition relocation occurs, 
> the newly created replica may have messages with old timestamp and cause the 
> log segment rolling for each message. The fix is to change the log rolling 
> behavior back to based on segment create time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

RE: Kafka consumers unable to process message

[Ghosh, Achintya (Contractor)]

I'm trying get the consumer logs and will send you. So it means it can happen 
even my local datacenter too. Still I'm not understanding if 3 nodes are up and 
message already replicated why it's trying to fetch the data from failed node. 
Can you please explain bit details how it works. Thanks for your response.

-Original Message-
From: Jason Gustafson [mailto:ja...@confluent.io] 
Sent: Wednesday, August 31, 2016 10:56 PM
To: us...@kafka.apache.org
Cc: dev@kafka.apache.org
Subject: Re: Kafka consumers unable to process message

The exceptions show one of the replica fetcher threads on the broker failing 
which makes perfect sense since some of the partitions were bound to have 
leaders in the failed datacenter. I'd actually like to see the consumer logs at 
DEBUG level if possible.

Thanks,
Jason

On Wed, Aug 31, 2016 at 7:48 PM, Ghosh, Achintya (Contractor) < 
achintya_gh...@comcast.com> wrote:

> Hi Jason,
>
> No, I didn't bring down any zookeeper server. Even I tried with 3 
> zookeeper server one as an 'Observer' but the same issue.
>
> Here is the server log from one of the node of my other datacenter:
>
> [2016-09-01 01:25:19,221] INFO Truncating log TEST3-0 to offset 0.
> (kafka.log.Log)
> [2016-09-01 01:25:19,257] INFO [ReplicaFetcherThread-0-3], Starting
> (kafka.server.ReplicaFetcherThread)
> [2016-09-01 01:25:19,258] INFO [ReplicaFetcherManager on broker 4] 
> Added fetcher for partitions List([[TEST3,0], initOffset 0 to broker 
> BrokerEndPoint(3,psaq3-wc.sys.comcast.net,61616)] ) (kafka.server.
> ReplicaFetcherManager)
> [2016-09-01 01:26:14,154] WARN [ReplicaFetcherThread-0-3], Error in 
> fetch
> kafka.server.ReplicaFetcherThread$FetchRequest@6618a925 (kafka.server.
> ReplicaFetcherThread)
> java.io.IOException: Connection to 3 was disconnected before the 
> response was read
> at kafka.utils.NetworkClientBlockingOps$$anonfun$
> blockingSendAndReceive$extension$1$$anonfun$apply$1.apply(
> NetworkClientBlockingOps.scala:87)
> at kafka.utils.NetworkClientBlockingOps$$anonfun$
> blockingSendAndReceive$extension$1$$anonfun$apply$1.apply(
> NetworkClientBlockingOps.scala:84)
> at scala.Option.foreach(Option.scala:257)
> at kafka.utils.NetworkClientBlockingOps$$anonfun$
> blockingSendAndReceive$extension$1.apply(NetworkClientBlockingOps.
> scala:84)
> at kafka.utils.NetworkClientBlockingOps$$anonfun$
> blockingSendAndReceive$extension$1.apply(NetworkClientBlockingOps.
> scala:80)
> at kafka.utils.NetworkClientBlockingOps$.recursivePoll$2(
> NetworkClientBlockingOps.scala:137)
> at kafka.utils.NetworkClientBlockingOps$.kafka$utils$
> NetworkClientBlockingOps$$pollContinuously$extension(
> NetworkClientBlockingOps.scala:143)
> at 
> kafka.utils.NetworkClientBlockingOps$.blockingSendAndReceive$
> extension(NetworkClientBlockingOps.scala:80)
> at kafka.server.ReplicaFetcherThread.sendRequest(
> ReplicaFetcherThread.scala:244)
> at kafka.server.ReplicaFetcherThread.fetch(
> ReplicaFetcherThread.scala:229)
> at kafka.server.ReplicaFetcherThread.fetch(
> ReplicaFetcherThread.scala:42)
> at kafka.server.AbstractFetcherThread.processFetchRequest(
> AbstractFetcherThread.scala:107)
> at kafka.server.AbstractFetcherThread.doWork(
> AbstractFetcherThread.scala:98)
> at 
> kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
> [2016-09-01 01:26:16,189] WARN [ReplicaFetcherThread-0-3], Error in 
> fetch
> kafka.server.ReplicaFetcherThread$FetchRequest@6e7e2578 (kafka.server.
> ReplicaFetcherThread)
> java.io.IOException: Connection to psaq3-wc.sys.comcast.net:61616 (id: 
> 3
> rack: null) failed
> at 
> kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$
> extension$2.apply(NetworkClientBlockingOps.scala:63)
> at 
> kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$
> extension$2.apply(NetworkClientBlockingOps.scala:59)
> at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(
> NetworkClientBlockingOps.scala:112)
> at kafka.utils.NetworkClientBlockingOps$.kafka$utils$
> NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.
> scala:120)
> at 
> kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(
> NetworkClientBlockingOps.scala:59)
> at kafka.server.ReplicaFetcherThread.sendRequest(
> ReplicaFetcherThread.scala:239)
> at kafka.server.ReplicaFetcherThread.fetch(
> ReplicaFetcherThread.scala:229)
> at kafka.server.ReplicaFetcherThread.fetch(
> ReplicaFetcherThread.scala:42)
> at kafka.server.AbstractFetcherThread.processFetchRequest(
> AbstractFetcherThread.scala:107)
> at kafka.server.AbstractFetcherThread.doWork(
> AbstractFetcherThread.scala:98)
> at 
> kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
> [2016-09-01 01:26:18,198] WARN [ReplicaFetcherThread-0-3], Error in 
> fetch 

Re: Kafka consumers unable to process message

[Jason Gustafson]

The exceptions show one of the replica fetcher threads on the broker
failing which makes perfect sense since some of the partitions were bound
to have leaders in the failed datacenter. I'd actually like to see the
consumer logs at DEBUG level if possible.

Thanks,
Jason

On Wed, Aug 31, 2016 at 7:48 PM, Ghosh, Achintya (Contractor) <
achintya_gh...@comcast.com> wrote:

> Hi Jason,
>
> No, I didn't bring down any zookeeper server. Even I tried with 3
> zookeeper server one as an 'Observer' but the same issue.
>
> Here is the server log from one of the node of my other datacenter:
>
> [2016-09-01 01:25:19,221] INFO Truncating log TEST3-0 to offset 0.
> (kafka.log.Log)
> [2016-09-01 01:25:19,257] INFO [ReplicaFetcherThread-0-3], Starting
> (kafka.server.ReplicaFetcherThread)
> [2016-09-01 01:25:19,258] INFO [ReplicaFetcherManager on broker 4] Added
> fetcher for partitions List([[TEST3,0], initOffset 0 to broker
> BrokerEndPoint(3,psaq3-wc.sys.comcast.net,61616)] ) (kafka.server.
> ReplicaFetcherManager)
> [2016-09-01 01:26:14,154] WARN [ReplicaFetcherThread-0-3], Error in fetch
> kafka.server.ReplicaFetcherThread$FetchRequest@6618a925 (kafka.server.
> ReplicaFetcherThread)
> java.io.IOException: Connection to 3 was disconnected before the response
> was read
> at kafka.utils.NetworkClientBlockingOps$$anonfun$
> blockingSendAndReceive$extension$1$$anonfun$apply$1.apply(
> NetworkClientBlockingOps.scala:87)
> at kafka.utils.NetworkClientBlockingOps$$anonfun$
> blockingSendAndReceive$extension$1$$anonfun$apply$1.apply(
> NetworkClientBlockingOps.scala:84)
> at scala.Option.foreach(Option.scala:257)
> at kafka.utils.NetworkClientBlockingOps$$anonfun$
> blockingSendAndReceive$extension$1.apply(NetworkClientBlockingOps.
> scala:84)
> at kafka.utils.NetworkClientBlockingOps$$anonfun$
> blockingSendAndReceive$extension$1.apply(NetworkClientBlockingOps.
> scala:80)
> at kafka.utils.NetworkClientBlockingOps$.recursivePoll$2(
> NetworkClientBlockingOps.scala:137)
> at kafka.utils.NetworkClientBlockingOps$.kafka$utils$
> NetworkClientBlockingOps$$pollContinuously$extension(
> NetworkClientBlockingOps.scala:143)
> at kafka.utils.NetworkClientBlockingOps$.blockingSendAndReceive$
> extension(NetworkClientBlockingOps.scala:80)
> at kafka.server.ReplicaFetcherThread.sendRequest(
> ReplicaFetcherThread.scala:244)
> at kafka.server.ReplicaFetcherThread.fetch(
> ReplicaFetcherThread.scala:229)
> at kafka.server.ReplicaFetcherThread.fetch(
> ReplicaFetcherThread.scala:42)
> at kafka.server.AbstractFetcherThread.processFetchRequest(
> AbstractFetcherThread.scala:107)
> at kafka.server.AbstractFetcherThread.doWork(
> AbstractFetcherThread.scala:98)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
> [2016-09-01 01:26:16,189] WARN [ReplicaFetcherThread-0-3], Error in fetch
> kafka.server.ReplicaFetcherThread$FetchRequest@6e7e2578 (kafka.server.
> ReplicaFetcherThread)
> java.io.IOException: Connection to psaq3-wc.sys.comcast.net:61616 (id: 3
> rack: null) failed
> at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$
> extension$2.apply(NetworkClientBlockingOps.scala:63)
> at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$
> extension$2.apply(NetworkClientBlockingOps.scala:59)
> at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(
> NetworkClientBlockingOps.scala:112)
> at kafka.utils.NetworkClientBlockingOps$.kafka$utils$
> NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.
> scala:120)
> at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(
> NetworkClientBlockingOps.scala:59)
> at kafka.server.ReplicaFetcherThread.sendRequest(
> ReplicaFetcherThread.scala:239)
> at kafka.server.ReplicaFetcherThread.fetch(
> ReplicaFetcherThread.scala:229)
> at kafka.server.ReplicaFetcherThread.fetch(
> ReplicaFetcherThread.scala:42)
> at kafka.server.AbstractFetcherThread.processFetchRequest(
> AbstractFetcherThread.scala:107)
> at kafka.server.AbstractFetcherThread.doWork(
> AbstractFetcherThread.scala:98)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
> [2016-09-01 01:26:18,198] WARN [ReplicaFetcherThread-0-3], Error in fetch
> kafka.server.ReplicaFetcherThread$FetchRequest@5adea8fb (kafka.server.
> ReplicaFetcherThread)
> java.io.IOException: Connection to psaq3-wc.sys.comcast.net:61616 (id: 3
> rack: null) failed
> at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$
> extension$2.apply(NetworkClientBlockingOps.scala:63)
> at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$
> extension$2.apply(NetworkClientBlockingOps.scala:59)
> at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(
> NetworkClientBlockingOps.scala:112)
> at kafka.utils.NetworkClientBlockingOps$.kafka$utils$
> 

RE: Kafka consumers unable to process message

[Ghosh, Achintya (Contractor)]

Hi Jason,

No, I didn't bring down any zookeeper server. Even I tried with 3 zookeeper 
server one as an 'Observer' but the same issue.

Here is the server log from one of the node of my other datacenter:

[2016-09-01 01:25:19,221] INFO Truncating log TEST3-0 to offset 0. 
(kafka.log.Log)
[2016-09-01 01:25:19,257] INFO [ReplicaFetcherThread-0-3], Starting  
(kafka.server.ReplicaFetcherThread)
[2016-09-01 01:25:19,258] INFO [ReplicaFetcherManager on broker 4] Added 
fetcher for partitions List([[TEST3,0], initOffset 0 to broker 
BrokerEndPoint(3,psaq3-wc.sys.comcast.net,61616)] ) 
(kafka.server.ReplicaFetcherManager)
[2016-09-01 01:26:14,154] WARN [ReplicaFetcherThread-0-3], Error in fetch 
kafka.server.ReplicaFetcherThread$FetchRequest@6618a925 
(kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to 3 was disconnected before the response was 
read
at 
kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1$$anonfun$apply$1.apply(NetworkClientBlockingOps.scala:87)
at 
kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1$$anonfun$apply$1.apply(NetworkClientBlockingOps.scala:84)
at scala.Option.foreach(Option.scala:257)
at 
kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1.apply(NetworkClientBlockingOps.scala:84)
at 
kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1.apply(NetworkClientBlockingOps.scala:80)
at 
kafka.utils.NetworkClientBlockingOps$.recursivePoll$2(NetworkClientBlockingOps.scala:137)
at 
kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollContinuously$extension(NetworkClientBlockingOps.scala:143)
at 
kafka.utils.NetworkClientBlockingOps$.blockingSendAndReceive$extension(NetworkClientBlockingOps.scala:80)
at 
kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:244)
at 
kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
at 
kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
at 
kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
at 
kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2016-09-01 01:26:16,189] WARN [ReplicaFetcherThread-0-3], Error in fetch 
kafka.server.ReplicaFetcherThread$FetchRequest@6e7e2578 
(kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to psaq3-wc.sys.comcast.net:61616 (id: 3 rack: 
null) failed
at 
kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
at 
kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
at 
kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
at 
kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
at 
kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
at 
kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:239)
at 
kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
at 
kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
at 
kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
at 
kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2016-09-01 01:26:18,198] WARN [ReplicaFetcherThread-0-3], Error in fetch 
kafka.server.ReplicaFetcherThread$FetchRequest@5adea8fb 
(kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to psaq3-wc.sys.comcast.net:61616 (id: 3 rack: 
null) failed
at 
kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
at 
kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
at 
kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
at 
kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
at 
kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
at 
kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:239)
at 
kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
at 
kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
at 
kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
   

Re: Kafka consumers unable to process message

[Jason Gustafson]

Hi Achintya,

Just to clarify, you did not take down either of the zookeepers in this
test, right? Having only two zookeepers in the ensemble would mean that if
either one of them failed, zookeeper wouldn't be able to reach quorum.

I'm not entirely sure why this would happen. One possibility is that the
consumer is failing to find the new coordinator, which might happen if all
the replicas for one of the __consumer_offsets partitions were located in
the "failed" datacenter. Perhaps you can enable DEBUG logging and post some
logs so we can see what it's actually doing during poll().

By the way, I noticed that your consumer configuration settings seem a
little mixed up. The new consumer doesn't actually communicate with
Zookeeper, so there's no need for those settings. And you don't need to
include the "offsets.storage" option since Kafka is the only choice. Also,
I don't think "consumer.timeout.ms" is an option.

-Jason


On Wed, Aug 31, 2016 at 6:43 PM, Ghosh, Achintya (Contractor) <
achintya_gh...@comcast.com> wrote:

> Hi Jason,
>
> Thanks for your response.
>
> I know that is a known issue and I resolved it calling wakeup method by
> another thread. But here my problem is different, let me explain , it's
> very basic
>
> I created one cluster with 6 nodes( 3 from one datacenter and 3 from
> another(remote) datacenter and kept replication factor 6 with 2 zookeeper
> servers one from each datacenter ). Now I brought down all 3 nodes of my
> local datacenter and produced few messages and I see producer is working
> fine even my local data center nodes are down. It successfully writes the
> messages to other data center nodes. But when I'm trying to consume the
> messages the consumer.poll method gets stuck as my local datacenter is down
> though other datacenter's nodes are up.
>
> My question is as the data has been written successfully to other
> datacenter why consumer part is not working?
>
> Here is my Producer settings:
>
> props.put("bootstrap.servers", "psaq1-wc.sys.comcast.net:61616,
> psaq2-wc.sys.comcast.net:61616,psaq3-wc.sys.comcast.net:61616,psaq1-ab.
> sys.comcast.net:61617,psaq2-ab.sys.comcast.net:61617,psaq3
> -ab.sys.comcast.net:61617");
> props.put("acks", "1");
> props.put("max.block.ms", 1000);
> props.put("key.serializer", "org.apache.kafka.common.serialization.
> StringSerializer");
> props.put("value.serializer", "com.comcast.ps.kafka.object.
> CustomMessageSer");
>
> and here is Consumer settings:
>
> props.put("group.id", "app-consumer");
> props.put("enable.auto.commit", "false");
> props.put("auto.offset.reset", "earliest");
> props.put("auto.commit.interval.ms", "500");
> props.put("session.timeout.ms", "12");
> props.put("consumer.timeout.ms", "1");
> props.put("zookeeper.session.timeout.ms", "12");
> props.put("zookeeper.connection.timeout.ms", "6");
> props.put("offsets.storage","kafka");
> props.put("request.timeout.ms", "15");
> props.put("bootstrap.servers", "psaq1-wc.sys.comcast.net:
> 61616,psaq2-wc.sys.comcast.net:61616,psaq3-wc.sys.comcast.net:61616,
> psaq1-ab.sys.comcast.net:61617,psaq2-ab.sys.comcast.net:61617,psaq3
> -ab.sys.comcast.net:61617");
> props.put("key.deserializer", "org.apache.kafka.common.
> serialization.StringDeserializer");
> props.put("value.deserializer",
> "com.comcast.ps.kafka.object.CustomMessageDeSer");
>
> Is it because of consumer is not able to get the broker metadata if it is
> trying to connect other datacenter's zookeeper server? I tried with to
> increate the zookeeper session timeout and connection time out but no luck.
>
> Please help on this.
> Thanks
> Achintya
>
>
> -Original Message-
> From: Jason Gustafson [mailto:ja...@confluent.io]
> Sent: Wednesday, August 31, 2016 4:05 PM
> To: us...@kafka.apache.org
> Cc: dev@kafka.apache.org
> Subject: Re: Kafka consumers unable to process message
>
> Hi Achintya,
>
> We have a JIRA for this problem: https://issues.
> apache.org/jira/browse/KAFKA-3834. Do you expect the client to raise an
> exception in this case or do you just want to keep it from blocking
> indefinitely? If the latter, you could escape the poll from another thread
> using wakeup().
>
> Thanks,
> Jason
>
> On Wed, Aug 31, 2016 at 12:11 PM, Ghosh, Achintya (Contractor) <
> achintya_gh...@comcast.com> wrote:
>
> > Hi there,
> >
> > Kafka consumer gets stuck at consumer.poll() method if my current
> > datacenter is down and replicated messages are in remote datacenter.
> >
> > How to solve that issue?
> >
> > Thanks
> > Achintya
> >
>

[jira] [Updated] (KAFKA-4109) kafka client send msg exception

[frank (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

frank updated KAFKA-4109:
-
Environment: 
java8
kafka cluster

  was:java8


> kafka client send msg exception
> ---
>
> Key: KAFKA-4109
> URL: https://issues.apache.org/jira/browse/KAFKA-4109
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 0.10.0.0
> Environment: java8
> kafka cluster
>Reporter: frank
>   Original Estimate: 5m
>  Remaining Estimate: 5m
>
> java.util.concurrent.ExecutionException: 
> org.apache.kafka.common.errors.TimeoutException: Batch Expired 
>   at 
> org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:56)
>  
>   at 
> org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:43)
>  
>   at 
> org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:25)
>  
>   at 
> com.longsheng.basicCollect.kafka.KafkaProducer.publishMessage(KafkaProducer.java:44)
>  
>   at 
> com.longsheng.basicCollect.crawler.processor.dzwbigdata.ParentBasic.publish(ParentBasic.java:60)
>  
>   at 
> com.longsheng.basicCollect.crawler.processor.dzwbigdata.ParentBasic.parseIncJson(ParentBasic.java:119)
>  
>   at 
> com.longsheng.basicCollect.crawler.processor.dzwbigdata.coke.price.SecondPrice.collectData(SecondPrice.java:41)
>  
>   at 
> com.longsheng.basicCollect.crawler.processor.dzwbigdata.coke.price.SecondPrice.process(SecondPrice.java:49)
>  
>   at 
> com.longsheng.basicCollect.crawler.processor.dzwbigdata.DzwProcessor.process(DzwProcessor.java:33)
>  
>   at 
> com.longsheng.basicCollect.timer.CollectDzwBigData.execute(CollectDzwBigData.java:14)
>  
>   at org.quartz.core.JobRunShell.run(JobRunShell.java:202) 
>   at 
> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) 
> Caused by: org.apache.kafka.common.errors.TimeoutException: Batch Expired
> the exception is arbitrarily!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-4109) kafka client send msg exception

[frank (JIRA)]

frank created KAFKA-4109:


 Summary: kafka client send msg exception
 Key: KAFKA-4109
 URL: https://issues.apache.org/jira/browse/KAFKA-4109
 Project: Kafka
  Issue Type: Bug
  Components: producer 
Affects Versions: 0.10.0.0
 Environment: java8
Reporter: frank


java.util.concurrent.ExecutionException: 
org.apache.kafka.common.errors.TimeoutException: Batch Expired 
at 
org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:56)
 
at 
org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:43)
 
at 
org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:25)
 
at 
com.longsheng.basicCollect.kafka.KafkaProducer.publishMessage(KafkaProducer.java:44)
 
at 
com.longsheng.basicCollect.crawler.processor.dzwbigdata.ParentBasic.publish(ParentBasic.java:60)
 
at 
com.longsheng.basicCollect.crawler.processor.dzwbigdata.ParentBasic.parseIncJson(ParentBasic.java:119)
 
at 
com.longsheng.basicCollect.crawler.processor.dzwbigdata.coke.price.SecondPrice.collectData(SecondPrice.java:41)
 
at 
com.longsheng.basicCollect.crawler.processor.dzwbigdata.coke.price.SecondPrice.process(SecondPrice.java:49)
 
at 
com.longsheng.basicCollect.crawler.processor.dzwbigdata.DzwProcessor.process(DzwProcessor.java:33)
 
at 
com.longsheng.basicCollect.timer.CollectDzwBigData.execute(CollectDzwBigData.java:14)
 
at org.quartz.core.JobRunShell.run(JobRunShell.java:202) 
at 
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) 
Caused by: org.apache.kafka.common.errors.TimeoutException: Batch Expired




the exception is arbitrarily!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

RE: Kafka consumers unable to process message

[Ghosh, Achintya (Contractor)]

Hi Jason,

Thanks for your response.

I know that is a known issue and I resolved it calling wakeup method by another 
thread. But here my problem is different, let me explain , it's very basic

I created one cluster with 6 nodes( 3 from one datacenter and 3 from 
another(remote) datacenter and kept replication factor 6 with 2 zookeeper 
servers one from each datacenter ). Now I brought down all 3 nodes of my local 
datacenter and produced few messages and I see producer is working fine even my 
local data center nodes are down. It successfully writes the messages to other 
data center nodes. But when I'm trying to consume the messages the 
consumer.poll method gets stuck as my local datacenter is down though other 
datacenter's nodes are up. 

My question is as the data has been written successfully to other datacenter 
why consumer part is not working?

Here is my Producer settings:

props.put("bootstrap.servers", 
"psaq1-wc.sys.comcast.net:61616,psaq2-wc.sys.comcast.net:61616,psaq3-wc.sys.comcast.net:61616,psaq1-ab.sys.comcast.net:61617,psaq2-ab.sys.comcast.net:61617,psaq3-ab.sys.comcast.net:61617");
props.put("acks", "1");
props.put("max.block.ms", 1000);
props.put("key.serializer", 
"org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "com.comcast.ps.kafka.object.CustomMessageSer");

and here is Consumer settings:

props.put("group.id", "app-consumer");
props.put("enable.auto.commit", "false");
props.put("auto.offset.reset", "earliest");
props.put("auto.commit.interval.ms", "500");
props.put("session.timeout.ms", "12");
props.put("consumer.timeout.ms", "1");
props.put("zookeeper.session.timeout.ms", "12");
props.put("zookeeper.connection.timeout.ms", "6");
props.put("offsets.storage","kafka");
props.put("request.timeout.ms", "15");
props.put("bootstrap.servers", 
"psaq1-wc.sys.comcast.net:61616,psaq2-wc.sys.comcast.net:61616,psaq3-wc.sys.comcast.net:61616,psaq1-ab.sys.comcast.net:61617,psaq2-ab.sys.comcast.net:61617,psaq3-ab.sys.comcast.net:61617");

props.put("key.deserializer", 
"org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", 
"com.comcast.ps.kafka.object.CustomMessageDeSer");

Is it because of consumer is not able to get the broker metadata if it is 
trying to connect other datacenter's zookeeper server? I tried with to increate 
the zookeeper session timeout and connection time out but no luck.

Please help on this.
Thanks
Achintya


-Original Message-
From: Jason Gustafson [mailto:ja...@confluent.io] 
Sent: Wednesday, August 31, 2016 4:05 PM
To: us...@kafka.apache.org
Cc: dev@kafka.apache.org
Subject: Re: Kafka consumers unable to process message

Hi Achintya,

We have a JIRA for this problem: https://issues.
apache.org/jira/browse/KAFKA-3834. Do you expect the client to raise an 
exception in this case or do you just want to keep it from blocking 
indefinitely? If the latter, you could escape the poll from another thread 
using wakeup().

Thanks,
Jason

On Wed, Aug 31, 2016 at 12:11 PM, Ghosh, Achintya (Contractor) < 
achintya_gh...@comcast.com> wrote:

> Hi there,
>
> Kafka consumer gets stuck at consumer.poll() method if my current 
> datacenter is down and replicated messages are in remote datacenter.
>
> How to solve that issue?
>
> Thanks
> Achintya
>

Re: Kafka consumers unable to process message

[Jason Gustafson]

Kafka clients have tended to make broker retries transparent to the user.
There's been discussion on various JIRAs about what we should do when all
the known brokers become unreachable. One option is to revert to the
configured bootstrap broker list, which is nice if you've configured a vip
for bootstrapping. More generally, we've discussed introducing a pluggable
interface for broker discovery, which allows for integration with service
discovery frameworks like consul. I'm supportive of this option, but we
probably need a champion with a little more time to investigate the options
and push it through. For the JIRA that I linked to above, I'm inclined to
have poll() silently retry since that is consistent with current behavior,
but it should not block longer than the passed timeout.

-Jason

On Wed, Aug 31, 2016 at 3:00 PM, Jim Jagielski  wrote:

> Yeah, let's figure out the "best" action to take...
>
> Looks like something I'd like to get a handle on.
>
> > On Aug 31, 2016, at 4:05 PM, Jason Gustafson  wrote:
> >
> > Hi Achintya,
> >
> > We have a JIRA for this problem: https://issues.
> > apache.org/jira/browse/KAFKA-3834. Do you expect the client to raise an
> > exception in this case or do you just want to keep it from blocking
> > indefinitely? If the latter, you could escape the poll from another
> thread
> > using wakeup().
> >
> > Thanks,
> > Jason
> >
> > On Wed, Aug 31, 2016 at 12:11 PM, Ghosh, Achintya (Contractor) <
> > achintya_gh...@comcast.com> wrote:
> >
> >> Hi there,
> >>
> >> Kafka consumer gets stuck at consumer.poll() method if my current
> >> datacenter is down and replicated messages are in remote datacenter.
> >>
> >> How to solve that issue?
> >>
> >> Thanks
> >> Achintya
> >>
>
>

Build failed in Jenkins: kafka-trunk-jdk8 #855

[Apache Jenkins Server]

See 

Changes:

[me] KAFKA-4103: Fix regression in DumpLogSegments offsets decoder

--
[...truncated 12258 lines...]
org.apache.kafka.streams.processor.internals.StreamTaskTest > testProcessOrder 
PASSED

org.apache.kafka.streams.processor.internals.StreamTaskTest > testPauseResume 
STARTED

org.apache.kafka.streams.processor.internals.StreamTaskTest > testPauseResume 
PASSED

org.apache.kafka.streams.processor.internals.StreamTaskTest > 
testMaybePunctuate STARTED

org.apache.kafka.streams.processor.internals.StreamTaskTest > 
testMaybePunctuate PASSED

org.apache.kafka.streams.processor.internals.assignment.AssignmentInfoTest > 
testEncodeDecode STARTED

org.apache.kafka.streams.processor.internals.assignment.AssignmentInfoTest > 
testEncodeDecode PASSED

org.apache.kafka.streams.processor.internals.assignment.AssignmentInfoTest > 
shouldDecodePreviousVersion STARTED

org.apache.kafka.streams.processor.internals.assignment.AssignmentInfoTest > 
shouldDecodePreviousVersion PASSED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
testEncodeDecode STARTED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
testEncodeDecode PASSED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
shouldEncodeDecodeWithUserEndPoint STARTED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
shouldEncodeDecodeWithUserEndPoint PASSED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
shouldBeBackwardCompatible STARTED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
shouldBeBackwardCompatible PASSED

org.apache.kafka.streams.processor.internals.assignment.TaskAssignorTest > 
testStickiness STARTED

org.apache.kafka.streams.processor.internals.assignment.TaskAssignorTest > 
testStickiness PASSED

org.apache.kafka.streams.processor.internals.assignment.TaskAssignorTest > 
testAssignWithStandby STARTED

org.apache.kafka.streams.processor.internals.assignment.TaskAssignorTest > 
testAssignWithStandby PASSED

org.apache.kafka.streams.processor.internals.assignment.TaskAssignorTest > 
testAssignWithoutStandby STARTED

org.apache.kafka.streams.processor.internals.assignment.TaskAssignorTest > 
testAssignWithoutStandby PASSED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingMultiplexingTopology STARTED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingMultiplexingTopology PASSED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingStatefulTopology STARTED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingStatefulTopology PASSED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingSimpleTopology STARTED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingSimpleTopology PASSED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingSimpleMultiSourceTopology STARTED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingSimpleMultiSourceTopology PASSED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testTopologyMetadata STARTED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testTopologyMetadata PASSED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingMultiplexByNameTopology STARTED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingMultiplexByNameTopology PASSED

org.apache.kafka.streams.processor.internals.RecordCollectorTest > 
testSpecificPartition STARTED

org.apache.kafka.streams.processor.internals.RecordCollectorTest > 
testSpecificPartition PASSED

org.apache.kafka.streams.processor.internals.RecordCollectorTest > 
testStreamPartitioner STARTED

org.apache.kafka.streams.processor.internals.RecordCollectorTest > 
testStreamPartitioner PASSED

org.apache.kafka.streams.processor.internals.PunctuationQueueTest > 
testPunctuationInterval STARTED

org.apache.kafka.streams.processor.internals.PunctuationQueueTest > 
testPunctuationInterval PASSED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
shouldThrowExceptionIfApplicationServerConfigIsNotHostPortPair STARTED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
shouldThrowExceptionIfApplicationServerConfigIsNotHostPortPair PASSED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
shouldMapUserEndPointToTopicPartitions STARTED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
shouldMapUserEndPointToTopicPartitions PASSED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
shouldAddUserDefinedEndPointToSubscription STARTED

[GitHub] kafka pull request #1808: MINOR: changes embedded broker time to MockTime

[mjsax]

GitHub user mjsax opened a pull request:

https://github.com/apache/kafka/pull/1808

MINOR: changes embedded broker time to MockTime



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mjsax/kafka mockTime

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1808.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1808






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] KIP-79 - ListOffsetRequest v1 and offsetForTime() method in new consumer.

[Becket Qin]

Hi Mayuresh,

Thanks for the comments.

1) I added the link to the Kafka protocol guide.

2) True that for LogAppendTime the message timestamp in different cluster
will not be the same. I updated the wiki. But notice that even the
LogAppendTime on different clusters are still comparable given a reasonable
delay bound. This use case was discussed in KIP-31/32.

3) TimestampOffset is just a simple container class of Timestamp and Offset.

Thanks,

Jiangjie (Becket) Qin

On Wed, Aug 31, 2016 at 1:38 PM, Mayuresh Gharat  wrote:

> Hi Becket,
>
> Thanks for the write up. I had few minor comments :
>
> 1) The wiki says ListOffsetRequest v0 still behaves in the old way.
> ---> It would be good to include what it means although that's not the part
> of this KIP.
> You can add a note what V0 means like :
> (Kafka allows querying offsets of messages by time and it does so at
> segment granularity. The timestamp parameter is the unix timestamp and
> querying the offset by timestamp returns the latest possible offset of the
> message that is appended no later than the given timestamp. There are 2
> special values of the timestamp - latest and earliest. For any other value
> of the unix timestamp, Kafka will get the starting offset of the log
> segment that is created no later than the given timestamp. Due to this, and
> since the offset request is served only at segment granularity, the offset
> fetch request returns less accurate results for larger segment sizes.)
>
> This will be easier for first time reader to just understand the difference
> between old and new without having to dig through code or refer online
> docs.
>
>
> 2) For the point :
> In a multi-datacenter enviroment, users may have different Kafka clusters
> in each datacenter for disater recovery. If one of the datacenter failed,
> the applications may need to switch the consumption from one data center to
> another datacenter. Because the offset between two different Kafka clusters
> are independent, users cannot use the offsets from the failed datacenter to
> consume from the DR datacenter. In this case, searching by timestamp will
> help because the messages should have the same timestamp.
> ---> This may not be true always right because it depends upon how the
> timestamp is created like if its the create time of the message or the file
> append time. So if its file append time, it will be different on two
> different broker machines right? In that case the offsets for the same
> timestamp will be different on the two broker machines right?
>
> 3) Can describe how the “TimestampOffset” class looks like?
>
>
> Nice wiki !
>
> Thanks,
>
> Mayuresh
>
> On Tue, Aug 30, 2016 at 8:38 PM, Becket Qin  wrote:
>
> > Hi Kafka devs,
> >
> > I created KIP-79 to allow consumer to precisely query the offsets based
> on
> > timestamp.
> >
> > In short we propose to :
> > 1. add a ListOffsetRequest/ListOffsetResponse v1, and
> > 2. add an offsetForTime() method in new consumer.
> >
> > The KIP wiki is the following:
> > https://cwiki.apache.org/confluence/pages/viewpage.
> action?pageId=65868090
> >
> > Comments are welcome.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
>
>
>
> --
> -Regards,
> Mayuresh R. Gharat
> (862) 250-7125
>

Re: Kafka consumers unable to process message

[Jim Jagielski]

Yeah, let's figure out the "best" action to take...

Looks like something I'd like to get a handle on.

> On Aug 31, 2016, at 4:05 PM, Jason Gustafson  wrote:
> 
> Hi Achintya,
> 
> We have a JIRA for this problem: https://issues.
> apache.org/jira/browse/KAFKA-3834. Do you expect the client to raise an
> exception in this case or do you just want to keep it from blocking
> indefinitely? If the latter, you could escape the poll from another thread
> using wakeup().
> 
> Thanks,
> Jason
> 
> On Wed, Aug 31, 2016 at 12:11 PM, Ghosh, Achintya (Contractor) <
> achintya_gh...@comcast.com> wrote:
> 
>> Hi there,
>> 
>> Kafka consumer gets stuck at consumer.poll() method if my current
>> datacenter is down and replicated messages are in remote datacenter.
>> 
>> How to solve that issue?
>> 
>> Thanks
>> Achintya
>> 

Re: [VOTE] KIP-33 - Add a time based log index

[Becket Qin]

Thanks for the feedback, Guozhang. I'll update the KIP wiki and submit a
patch if no one have concerns on this.

On Wed, Aug 31, 2016 at 11:59 AM, Guozhang Wang  wrote:

> Some of the streams integration tests also encounters this issue where the
> timestamps we are using in the test are very small (e.g. 1,2,3...) which
> cause the log to roll upon each append, and the old segment gets deleted
> very soon. Arguably this can be resolved to enforce LogAppendTime
> configuration on the embedded server.
>
> +1 on the proposed change, makes sense to me.
>
> Guozhang
>
>
> On Tue, Aug 30, 2016 at 4:33 PM, Becket Qin  wrote:
>
> > Hi folks,
> >
> > Here is another update on the change of time based log rolling.
> >
> > After the latest implementation, we encountered KAFKA-4099. The issue is
> > that if users move replicas, for the messages in the old segments, the
> new
> > replica will create one log segment for each message. The root cause of
> > this is we are comparing the wall clock time with the message timestamp.
> A
> > solution to that is also described in KAFKA-4099, which is to change the
> > log rolling purely based on the timestamp in the messages. More
> > specifically, we roll out the log segment if the timestamp in the current
> > message is greater than the timestamp of the first message in the segment
> > by more than log.roll.ms. This approach is wall clock independent and
> > should solve the problem. With message.timestamp.difference.max.ms
> > configuration, we can achieve 1) the log segment will be rolled out in a
> > bounded time, 2) no excessively large timestamp will be accepted and
> cause
> > frequent log rolling.
> >
> > Any concern regarding this change?
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> > On Mon, Jun 13, 2016 at 2:30 PM, Guozhang Wang 
> wrote:
> >
> > > Thanks Jiangjie,
> > >
> > > I see the need for sensitive data purging, the above proposed change
> > LGTM.
> > > One minor concern is that a wrongly marked timestamp on the first
> record
> > > could cause the segment to roll much later / earlier, though it may be
> > > rare.
> > >
> > > Guozhang
> > >
> > > On Fri, Jun 10, 2016 at 10:07 AM, Becket Qin 
> > wrote:
> > >
> > > > Hi,
> > > >
> > > > During the implementation of KIP-33, we found it might be useful to
> > have
> > > a
> > > > more deterministic time based log rolling than what proposed in the
> > KIP.
> > > >
> > > > The current KIP proposal uses the largest timestamp in the segment
> for
> > > time
> > > > based rolling. The active log segment only rolls when there is no
> > message
> > > > appended in max.roll.ms since the largest timestamp in the segment.
> > i.e.
> > > > the rolling time may change if user keeping appending messages into
> the
> > > > segment. This may not be a desirable behavior for people who have
> > > sensitive
> > > > data and want to make sure they are removed after some time.
> > > >
> > > > To solve the above issue, we want to modify the KIP proposal
> regarding
> > > the
> > > > time based rolling to the following behavior. The time based log
> > rolling
> > > > will be based on the first message with a timestamp in the log
> segment
> > if
> > > > there is such a message. If no message in the segment has a
> timestamp,
> > > the
> > > > time based log rolling will still be based on log segment create
> time,
> > > > which is the same as we are doing now. The reasons we don't want to
> > > always
> > > > roll based on file create time are because 1) the message timestamp
> may
> > > be
> > > > assigned by clients which can be different from the create time of
> the
> > > log
> > > > segment file. 2) On some Linux, the file create time is not
> available,
> > so
> > > > using segment file create time may not always work.
> > > >
> > > > Do people have any concern for this change? I will update the KIP if
> > > people
> > > > think the change is OK.
> > > >
> > > > Thanks,
> > > >
> > > > Jiangjie (Becket) Qin
> > > >
> > > > On Tue, Apr 19, 2016 at 6:27 PM, Becket Qin 
> > > wrote:
> > > >
> > > > > Thanks Joel and Ismael. I just updated the KIP based on your
> > feedback.
> > > > >
> > > > > KIP-33 has passed with +4 (binding) and +2 (non-binding)
> > > > >
> > > > > Thanks everyone for the reading, feedback and voting!
> > > > >
> > > > > Jiangjie (Becket) Qin
> > > > >
> > > > > On Tue, Apr 19, 2016 at 5:25 PM, Ismael Juma 
> > > wrote:
> > > > >
> > > > >> Thanks Becket. I think it would be nice to update the KIP with
> > regards
> > > > to
> > > > >> point 3 and 4.
> > > > >>
> > > > >> In any case, +1 (non-binding)
> > > > >>
> > > > >> Ismael
> > > > >>
> > > > >> On Tue, Apr 19, 2016 at 2:03 AM, Becket Qin  >
> > > > wrote:
> > > > >>
> > > > >> > Thanks for the comments Ismael. Please see the replies inline.
> > > > >> >
> > > > >> > On Mon, Apr 18, 2016 at 6:50 AM, 

[jira] [Created] (KAFKA-4108) Improve DumpLogSegments offsets-decoder output format

[Jason Gustafson (JIRA)]

Jason Gustafson created KAFKA-4108:
--

 Summary: Improve DumpLogSegments offsets-decoder output format
 Key: KAFKA-4108
 URL: https://issues.apache.org/jira/browse/KAFKA-4108
 Project: Kafka
  Issue Type: Improvement
  Components: tools
Reporter: Jason Gustafson


When using the DumpLogSegments with the "--offsets-decoder" option (for 
consuming __consumer_offsets), the encoding of group metadata makes it a little 
difficult to identify individual fields. In particular, we use the following 
formatted string for group metadata: 
{code}
${protocolType}:${groupMetadata.protocol}:${groupMetadata.generationId}:${assignment}
{code}
Keys have a similar formatting. Most users are probably not going to know which 
field is which based only on the output, so it would be helpful to include 
field names. Maybe we could just output a JSON object?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-4103) DumpLogSegments cannot print data from offsets topic

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava resolved KAFKA-4103.
--
Resolution: Fixed

Issue resolved by pull request 1807
[https://github.com/apache/kafka/pull/1807]

> DumpLogSegments cannot print data from offsets topic
> 
>
> Key: KAFKA-4103
> URL: https://issues.apache.org/jira/browse/KAFKA-4103
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
>Reporter: Ewen Cheslack-Postava
>Assignee: Jason Gustafson
>Priority: Blocker
> Fix For: 0.10.1.0
>
>
> It looks like there's been a regression in the DumpLogSegments tool. I'm 
> marking it a blocker since it appears we can no longer dump offset 
> information from this tool, which makes it really hard to debug anything 
> related to __consumer_offsets.
> The 0.10.0 branch seems to work fine, but even with offsets log files 
> generated using only old formats (0.10.0 branch), the DumpLogSegments tool 
> from trunk (i.e. 0.10.1.0-SNAPSHOT with latest githash 
> b91eeac9438b8718c410045b0e9191296ebb536d as of reporting this) will cause the 
> exception below. This was found while doing some basic testing of KAFKA-4062.
> {quote}
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> offset: 0 position: 0 CreateTime: 1472615183913 isvalid: true payloadsize: 
> 199 magic: 1 compresscodec: NoCompressionCodec crc: 2036280914 keysize: 
> 26Exception in thread "main" java.util.IllegalFormatConversionException: x != 
> scala.math.BigInt
>   at 
> java.util.Formatter$FormatSpecifier.failConversion(Formatter.java:4045)
>   at 
> java.util.Formatter$FormatSpecifier.printInteger(Formatter.java:2748)
>   at 
> java.util.Formatter$FormatSpecifier.print(Formatter.java:2702)
>   at java.util.Formatter.format(Formatter.java:2488)
>   at java.util.Formatter.format(Formatter.java:2423)
>   at java.lang.String.format(String.java:2792)
>   at 
> kafka.tools.DumpLogSegments$OffsetsMessageParser.kafka$tools$DumpLogSegments$OffsetsMessageParser$$hex(DumpLogSegments.scala:240)
>   at 
> kafka.tools.DumpLogSegments$OffsetsMessageParser$$anonfun$3.apply(DumpLogSegments.scala:272)
>   at 
> kafka.tools.DumpLogSegments$OffsetsMessageParser$$anonfun$3.apply(DumpLogSegments.scala:262)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at scala.collection.immutable.List.foreach(List.scala:318)
>   at 
> scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
>   at 
> scala.collection.AbstractTraversable.map(Traversable.scala:105)
>   at 
> kafka.tools.DumpLogSegments$OffsetsMessageParser.parseGroupMetadata(DumpLogSegments.scala:262)
>   at 
> kafka.tools.DumpLogSegments$OffsetsMessageParser.parse(DumpLogSegments.scala:290)
>   at 
> kafka.tools.DumpLogSegments$$anonfun$kafka$tools$DumpLogSegments$$dumpLog$1$$anonfun$apply$3.apply(DumpLogSegments.scala:332)
>   at 
> kafka.tools.DumpLogSegments$$anonfun$kafka$tools$DumpLogSegments$$dumpLog$1$$anonfun$apply$3.apply(DumpLogSegments.scala:312)
>   at scala.collection.Iterator$class.foreach(Iterator.scala:727)
>   at 
> kafka.utils.IteratorTemplate.foreach(IteratorTemplate.scala:30)
>   at 
> kafka.tools.DumpLogSegments$$anonfun$kafka$tools$DumpLogSegments$$dumpLog$1.apply(DumpLogSegments.scala:312)
>   at 
> kafka.tools.DumpLogSegments$$anonfun$kafka$tools$DumpLogSegments$$dumpLog$1.apply(DumpLogSegments.scala:310)
>   at scala.collection.Iterator$class.foreach(Iterator.scala:727)
>   at 
> kafka.utils.IteratorTemplate.foreach(IteratorTemplate.scala:30)
>   at 
> kafka.tools.DumpLogSegments$.kafka$tools$DumpLogSegments$$dumpLog(DumpLogSegments.scala:310)
>   at 
> kafka.tools.DumpLogSegments$$anonfun$main$1.apply(DumpLogSegments.scala:96)
>   at 
> kafka.tools.DumpLogSegments$$anonfun$main$1.apply(DumpLogSegments.scala:92)
>   at 
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
>   at 
> scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
>   at kafka.tools.DumpLogSegments$.main(DumpLogSegments.scala:92)
>   at kafka.tools.DumpLogSegments.main(DumpLogSegments.scala)
> {quote}
> I haven't really dug in, but the source of the error is confusing since the 
> relevant string formatting code doesn't seem to have changed anytime 
> recently. It seems it might 

[jira] [Commented] (KAFKA-4103) DumpLogSegments cannot print data from offsets topic

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15453298#comment-15453298
 ] 

ASF GitHub Bot commented on KAFKA-4103:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1807


> DumpLogSegments cannot print data from offsets topic
> 
>
> Key: KAFKA-4103
> URL: https://issues.apache.org/jira/browse/KAFKA-4103
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
>Reporter: Ewen Cheslack-Postava
>Assignee: Jason Gustafson
>Priority: Blocker
> Fix For: 0.10.1.0
>
>
> It looks like there's been a regression in the DumpLogSegments tool. I'm 
> marking it a blocker since it appears we can no longer dump offset 
> information from this tool, which makes it really hard to debug anything 
> related to __consumer_offsets.
> The 0.10.0 branch seems to work fine, but even with offsets log files 
> generated using only old formats (0.10.0 branch), the DumpLogSegments tool 
> from trunk (i.e. 0.10.1.0-SNAPSHOT with latest githash 
> b91eeac9438b8718c410045b0e9191296ebb536d as of reporting this) will cause the 
> exception below. This was found while doing some basic testing of KAFKA-4062.
> {quote}
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> offset: 0 position: 0 CreateTime: 1472615183913 isvalid: true payloadsize: 
> 199 magic: 1 compresscodec: NoCompressionCodec crc: 2036280914 keysize: 
> 26Exception in thread "main" java.util.IllegalFormatConversionException: x != 
> scala.math.BigInt
>   at 
> java.util.Formatter$FormatSpecifier.failConversion(Formatter.java:4045)
>   at 
> java.util.Formatter$FormatSpecifier.printInteger(Formatter.java:2748)
>   at 
> java.util.Formatter$FormatSpecifier.print(Formatter.java:2702)
>   at java.util.Formatter.format(Formatter.java:2488)
>   at java.util.Formatter.format(Formatter.java:2423)
>   at java.lang.String.format(String.java:2792)
>   at 
> kafka.tools.DumpLogSegments$OffsetsMessageParser.kafka$tools$DumpLogSegments$OffsetsMessageParser$$hex(DumpLogSegments.scala:240)
>   at 
> kafka.tools.DumpLogSegments$OffsetsMessageParser$$anonfun$3.apply(DumpLogSegments.scala:272)
>   at 
> kafka.tools.DumpLogSegments$OffsetsMessageParser$$anonfun$3.apply(DumpLogSegments.scala:262)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at scala.collection.immutable.List.foreach(List.scala:318)
>   at 
> scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
>   at 
> scala.collection.AbstractTraversable.map(Traversable.scala:105)
>   at 
> kafka.tools.DumpLogSegments$OffsetsMessageParser.parseGroupMetadata(DumpLogSegments.scala:262)
>   at 
> kafka.tools.DumpLogSegments$OffsetsMessageParser.parse(DumpLogSegments.scala:290)
>   at 
> kafka.tools.DumpLogSegments$$anonfun$kafka$tools$DumpLogSegments$$dumpLog$1$$anonfun$apply$3.apply(DumpLogSegments.scala:332)
>   at 
> kafka.tools.DumpLogSegments$$anonfun$kafka$tools$DumpLogSegments$$dumpLog$1$$anonfun$apply$3.apply(DumpLogSegments.scala:312)
>   at scala.collection.Iterator$class.foreach(Iterator.scala:727)
>   at 
> kafka.utils.IteratorTemplate.foreach(IteratorTemplate.scala:30)
>   at 
> kafka.tools.DumpLogSegments$$anonfun$kafka$tools$DumpLogSegments$$dumpLog$1.apply(DumpLogSegments.scala:312)
>   at 
> kafka.tools.DumpLogSegments$$anonfun$kafka$tools$DumpLogSegments$$dumpLog$1.apply(DumpLogSegments.scala:310)
>   at scala.collection.Iterator$class.foreach(Iterator.scala:727)
>   at 
> kafka.utils.IteratorTemplate.foreach(IteratorTemplate.scala:30)
>   at 
> kafka.tools.DumpLogSegments$.kafka$tools$DumpLogSegments$$dumpLog(DumpLogSegments.scala:310)
>   at 
> kafka.tools.DumpLogSegments$$anonfun$main$1.apply(DumpLogSegments.scala:96)
>   at 
> kafka.tools.DumpLogSegments$$anonfun$main$1.apply(DumpLogSegments.scala:92)
>   at 
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
>   at 
> scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
>   at kafka.tools.DumpLogSegments$.main(DumpLogSegments.scala:92)
>   at kafka.tools.DumpLogSegments.main(DumpLogSegments.scala)
> {quote}
> I haven't really dug in, but the source of the error is confusing since the 
> relevant string formatting code doesn't seem to have changed anytime 
> recently. It 

[GitHub] kafka pull request #1807: KAFKA-4103: Fix regression in DumpLogSegments offs...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1807


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] KIP-79 - ListOffsetRequest v1 and offsetForTime() method in new consumer.

[Mayuresh Gharat]

Hi Becket,

Thanks for the write up. I had few minor comments :

1) The wiki says ListOffsetRequest v0 still behaves in the old way.
---> It would be good to include what it means although that's not the part
of this KIP.
You can add a note what V0 means like :
(Kafka allows querying offsets of messages by time and it does so at
segment granularity. The timestamp parameter is the unix timestamp and
querying the offset by timestamp returns the latest possible offset of the
message that is appended no later than the given timestamp. There are 2
special values of the timestamp - latest and earliest. For any other value
of the unix timestamp, Kafka will get the starting offset of the log
segment that is created no later than the given timestamp. Due to this, and
since the offset request is served only at segment granularity, the offset
fetch request returns less accurate results for larger segment sizes.)

This will be easier for first time reader to just understand the difference
between old and new without having to dig through code or refer online docs.


2) For the point :
In a multi-datacenter enviroment, users may have different Kafka clusters
in each datacenter for disater recovery. If one of the datacenter failed,
the applications may need to switch the consumption from one data center to
another datacenter. Because the offset between two different Kafka clusters
are independent, users cannot use the offsets from the failed datacenter to
consume from the DR datacenter. In this case, searching by timestamp will
help because the messages should have the same timestamp.
---> This may not be true always right because it depends upon how the
timestamp is created like if its the create time of the message or the file
append time. So if its file append time, it will be different on two
different broker machines right? In that case the offsets for the same
timestamp will be different on the two broker machines right?

3) Can describe how the “TimestampOffset” class looks like?


Nice wiki !

Thanks,

Mayuresh

On Tue, Aug 30, 2016 at 8:38 PM, Becket Qin  wrote:

> Hi Kafka devs,
>
> I created KIP-79 to allow consumer to precisely query the offsets based on
> timestamp.
>
> In short we propose to :
> 1. add a ListOffsetRequest/ListOffsetResponse v1, and
> 2. add an offsetForTime() method in new consumer.
>
> The KIP wiki is the following:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65868090
>
> Comments are welcome.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>



-- 
-Regards,
Mayuresh R. Gharat
(862) 250-7125

[jira] [Work started] (KAFKA-3807) OffsetValidationTest - transient failure on test_broker_rolling_bounce

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-3807 started by Jason Gustafson.
--
> OffsetValidationTest - transient failure on test_broker_rolling_bounce
> --
>
> Key: KAFKA-3807
> URL: https://issues.apache.org/jira/browse/KAFKA-3807
> Project: Kafka
>  Issue Type: Bug
>  Components: system tests
>Reporter: Geoff Anderson
>Assignee: Jason Gustafson
>
> {code}
> test_id:
> 2016-05-28--001.kafkatest.tests.client.consumer_test.OffsetValidationTest.test_broker_rolling_bounce
> status: FAIL
> run time:   3 minutes 8.042 seconds
> Broker rolling bounce caused 2 unexpected group rebalances
> Traceback (most recent call last):
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py",
>  line 106, in run_all_tests
> data = self.run_single_test()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py",
>  line 162, in run_single_test
> return self.current_test_context.function(self.current_test)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/tests/client/consumer_test.py",
>  line 108, in test_broker_rolling_bounce
> "Broker rolling bounce caused %d unexpected group rebalances" % 
> unexpected_rebalances
> AssertionError: Broker rolling bounce caused 2 unexpected group rebalances
> {code}
> http://confluent-kafka-system-test-results.s3-us-west-2.amazonaws.com/2016-05-28--001.1464455059--apache--trunk--7b7c4a7/report.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-3807) OffsetValidationTest - transient failure on test_broker_rolling_bounce

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson reassigned KAFKA-3807:
--

Assignee: Jason Gustafson

> OffsetValidationTest - transient failure on test_broker_rolling_bounce
> --
>
> Key: KAFKA-3807
> URL: https://issues.apache.org/jira/browse/KAFKA-3807
> Project: Kafka
>  Issue Type: Bug
>  Components: system tests
>Reporter: Geoff Anderson
>Assignee: Jason Gustafson
>
> {code}
> test_id:
> 2016-05-28--001.kafkatest.tests.client.consumer_test.OffsetValidationTest.test_broker_rolling_bounce
> status: FAIL
> run time:   3 minutes 8.042 seconds
> Broker rolling bounce caused 2 unexpected group rebalances
> Traceback (most recent call last):
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py",
>  line 106, in run_all_tests
> data = self.run_single_test()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py",
>  line 162, in run_single_test
> return self.current_test_context.function(self.current_test)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/tests/client/consumer_test.py",
>  line 108, in test_broker_rolling_bounce
> "Broker rolling bounce caused %d unexpected group rebalances" % 
> unexpected_rebalances
> AssertionError: Broker rolling bounce caused 2 unexpected group rebalances
> {code}
> http://confluent-kafka-system-test-results.s3-us-west-2.amazonaws.com/2016-05-28--001.1464455059--apache--trunk--7b7c4a7/report.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk8 #854

[Apache Jenkins Server]

See 

Changes:

[wangguoz] MINOR: onCompletion: metadata valid only if no exception

--
[...truncated 3469 lines...]
kafka.api.AuthorizerIntegrationTest > testCommitWithNoTopicAccess PASSED

kafka.api.AuthorizerIntegrationTest > testProduceWithNoTopicAccess STARTED

kafka.api.AuthorizerIntegrationTest > testProduceWithNoTopicAccess PASSED

kafka.api.AuthorizerIntegrationTest > testProduceWithTopicWrite STARTED

kafka.api.AuthorizerIntegrationTest > testProduceWithTopicWrite PASSED

kafka.api.AuthorizerIntegrationTest > testAuthorization STARTED

kafka.api.AuthorizerIntegrationTest > testAuthorization PASSED

kafka.api.AuthorizerIntegrationTest > testUnauthorizedDeleteWithDescribe STARTED

kafka.api.AuthorizerIntegrationTest > testUnauthorizedDeleteWithDescribe PASSED

kafka.api.AuthorizerIntegrationTest > testConsumeWithTopicAndGroupRead STARTED

kafka.api.AuthorizerIntegrationTest > testConsumeWithTopicAndGroupRead PASSED

kafka.api.AuthorizerIntegrationTest > testUnauthorizedDeleteWithoutDescribe 
STARTED

kafka.api.AuthorizerIntegrationTest > testUnauthorizedDeleteWithoutDescribe 
PASSED

kafka.api.AuthorizerIntegrationTest > testConsumeWithTopicWrite STARTED

kafka.api.AuthorizerIntegrationTest > testConsumeWithTopicWrite PASSED

kafka.api.AuthorizerIntegrationTest > testOffsetFetchWithNoGroupAccess STARTED

kafka.api.AuthorizerIntegrationTest > testOffsetFetchWithNoGroupAccess PASSED

kafka.api.AuthorizerIntegrationTest > testCommitWithNoAccess STARTED

kafka.api.AuthorizerIntegrationTest > testCommitWithNoAccess PASSED

kafka.api.AuthorizerIntegrationTest > testCommitWithNoGroupAccess STARTED

kafka.api.AuthorizerIntegrationTest > testCommitWithNoGroupAccess PASSED

kafka.api.AuthorizerIntegrationTest > testConsumeWithNoAccess STARTED

kafka.api.AuthorizerIntegrationTest > testConsumeWithNoAccess PASSED

kafka.api.AuthorizerIntegrationTest > testOffsetFetchWithTopicAndGroupRead 
STARTED

kafka.api.AuthorizerIntegrationTest > testOffsetFetchWithTopicAndGroupRead 
PASSED

kafka.api.AuthorizerIntegrationTest > testCommitWithTopicDescribe STARTED

kafka.api.AuthorizerIntegrationTest > testCommitWithTopicDescribe PASSED

kafka.api.AuthorizerIntegrationTest > testProduceWithTopicDescribe STARTED

kafka.api.AuthorizerIntegrationTest > testProduceWithTopicDescribe PASSED

kafka.api.AuthorizerIntegrationTest > testOffsetFetchTopicDescribe STARTED

kafka.api.AuthorizerIntegrationTest > testOffsetFetchTopicDescribe PASSED

kafka.api.AuthorizerIntegrationTest > testCommitWithTopicAndGroupRead STARTED

kafka.api.AuthorizerIntegrationTest > testCommitWithTopicAndGroupRead PASSED

kafka.api.AuthorizerIntegrationTest > 
testSimpleConsumeWithExplicitSeekAndNoGroupAccess STARTED

kafka.api.AuthorizerIntegrationTest > 
testSimpleConsumeWithExplicitSeekAndNoGroupAccess PASSED

kafka.api.SaslPlainPlaintextConsumerTest > testCoordinatorFailover STARTED

kafka.api.SaslPlainPlaintextConsumerTest > testCoordinatorFailover PASSED

kafka.api.SaslPlainPlaintextConsumerTest > testSimpleConsumption STARTED

kafka.api.SaslPlainPlaintextConsumerTest > testSimpleConsumption PASSED

kafka.api.RequestResponseSerializationTest > 
testSerializationAndDeserialization STARTED

kafka.api.RequestResponseSerializationTest > 
testSerializationAndDeserialization PASSED

kafka.api.RequestResponseSerializationTest > testFetchResponseVersion STARTED

kafka.api.RequestResponseSerializationTest > testFetchResponseVersion PASSED

kafka.api.RequestResponseSerializationTest > testProduceResponseVersion STARTED

kafka.api.RequestResponseSerializationTest > testProduceResponseVersion PASSED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > testNoConsumeAcl STARTED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > testNoConsumeAcl PASSED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > testProduceConsumeViaAssign 
STARTED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > testProduceConsumeViaAssign 
PASSED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > testNoProduceAcl STARTED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > testNoProduceAcl PASSED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > testNoGroupAcl STARTED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > testNoGroupAcl PASSED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > 
testProduceConsumeViaSubscribe STARTED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > 
testProduceConsumeViaSubscribe PASSED

kafka.api.SaslSslConsumerTest > testCoordinatorFailover STARTED

kafka.api.SaslSslConsumerTest > testCoordinatorFailover PASSED

kafka.api.SaslSslConsumerTest > testSimpleConsumption STARTED

kafka.api.SaslSslConsumerTest > testSimpleConsumption PASSED

kafka.api.test.ProducerCompressionTest > testCompression[0] STARTED

kafka.api.test.ProducerCompressionTest > testCompression[0] PASSED

kafka.api.test.ProducerCompressionTest > testCompression[1] STARTED

Re: Kafka consumers unable to process message

[Jason Gustafson]

Hi Achintya,

We have a JIRA for this problem: https://issues.
apache.org/jira/browse/KAFKA-3834. Do you expect the client to raise an
exception in this case or do you just want to keep it from blocking
indefinitely? If the latter, you could escape the poll from another thread
using wakeup().

Thanks,
Jason

On Wed, Aug 31, 2016 at 12:11 PM, Ghosh, Achintya (Contractor) <
achintya_gh...@comcast.com> wrote:

> Hi there,
>
> Kafka consumer gets stuck at consumer.poll() method if my current
> datacenter is down and replicated messages are in remote datacenter.
>
> How to solve that issue?
>
> Thanks
> Achintya
>

[jira] [Created] (KAFKA-4107) Support offset reset capability in Kafka Connect

[Jason Gustafson (JIRA)]

Jason Gustafson created KAFKA-4107:
--

 Summary: Support offset reset capability in Kafka Connect
 Key: KAFKA-4107
 URL: https://issues.apache.org/jira/browse/KAFKA-4107
 Project: Kafka
  Issue Type: Improvement
  Components: KafkaConnect
Reporter: Jason Gustafson
Assignee: Ewen Cheslack-Postava


It would be useful in some cases to be able to reset connector offsets. For 
example, if a topic in Kafka corresponding to a source database is accidentally 
deleted (or deleted because of corrupt data), an administrator may want to 
reset offsets and reproduce the log from the beginning. It may also be useful 
to have support for overriding offsets, but that seems like a less likely use 
case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4033) KIP-70: Revise Partition Assignment Semantics on New Consumer's Subscription Change

[Vahid Hashemian (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vahid Hashemian updated KAFKA-4033:
---
Status: Patch Available  (was: In Progress)

> KIP-70: Revise Partition Assignment Semantics on New Consumer's Subscription 
> Change
> ---
>
> Key: KAFKA-4033
> URL: https://issues.apache.org/jira/browse/KAFKA-4033
> Project: Kafka
>  Issue Type: Bug
>Reporter: Vahid Hashemian
>Assignee: Vahid Hashemian
>
> Modify the new consumer's implementation of topics subscribe and unsubscribe 
> interfaces so that they do not cause an immediate assignment update (this is 
> how the regex subscribe interface is implemented). Instead, the assignment 
> remains valid until it has been revoked in the next rebalance.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4106) Consumer / add configure method to PartitionAssignor interface

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15453098#comment-15453098
 ] 

Guozhang Wang commented on KAFKA-4106:
--

[~fhussonnois] {PartitionAssignor} extends {Configurable} and its {configure} 
function is auto triggered when the object is created.

> Consumer / add configure method to PartitionAssignor interface
> --
>
> Key: KAFKA-4106
> URL: https://issues.apache.org/jira/browse/KAFKA-4106
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients, consumer
>Affects Versions: 0.10.0.1
>Reporter: Florian Hussonnois
>Priority: Minor
>
> Currently, we can implement a custom PartitionAssignor which will forward 
> user data that will be used during the assignments protocol. For example, 
> data can be used to implement a rack-aware assignor
> However, currently we cannot dynamically configure a PartitionAssignor 
> instance.
> It would be nice to add a method configure(Map PartitionAssignor interface. Then, this method will be invoked by the 
> KafkaConsumer  on each assignor, as this is do for deserializers.
> The code modifications are pretty straight-forward but involve modifying the 
> public interface PartitionAssignor. Does that mean this JIRA needs a KIP ?
> I can contribute to that improvement.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3410) Unclean leader election and "Halting because log truncation is not allowed"

[James Cheng (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15453103#comment-15453103
 ] 

James Cheng commented on KAFKA-3410:


[~maysamyabandeh] Oh right, yeah, I didn't expect KAFKA-3924 to fix this issue, 
but [~guozhang] asked me to check.

>From the behavior, it sounds like KAFKA-3924 functioned as expected: a 
>controlled shutdown that (unfortunately) may deadlock.


> Unclean leader election and "Halting because log truncation is not allowed"
> ---
>
> Key: KAFKA-3410
> URL: https://issues.apache.org/jira/browse/KAFKA-3410
> Project: Kafka
>  Issue Type: Bug
>Reporter: James Cheng
>
> I ran into a scenario where one of my brokers would continually shutdown, 
> with the error message:
> [2016-02-25 00:29:39,236] FATAL [ReplicaFetcherThread-0-1], Halting because 
> log truncation is not allowed for topic test, Current leader 1's latest 
> offset 0 is less than replica 2's latest offset 151 
> (kafka.server.ReplicaFetcherThread)
> I managed to reproduce it with the following scenario:
> 1. Start broker1, with unclean.leader.election.enable=false
> 2. Start broker2, with unclean.leader.election.enable=false
> 3. Create topic, single partition, with replication-factor 2.
> 4. Write data to the topic.
> 5. At this point, both brokers are in the ISR. Broker1 is the partition 
> leader.
> 6. Ctrl-Z on broker2. (Simulates a GC pause or a slow network) Broker2 gets 
> dropped out of ISR. Broker1 is still the leader. I can still write data to 
> the partition.
> 7. Shutdown Broker1. Hard or controlled, doesn't matter.
> 8. rm -rf the log directory of broker1. (This simulates a disk replacement or 
> full hardware replacement)
> 9. Resume broker2. It attempts to connect to broker1, but doesn't succeed 
> because broker1 is down. At this point, the partition is offline. Can't write 
> to it.
> 10. Resume broker1. Broker1 resumes leadership of the topic. Broker2 attempts 
> to join ISR, and immediately halts with the error message:
> [2016-02-25 00:29:39,236] FATAL [ReplicaFetcherThread-0-1], Halting because 
> log truncation is not allowed for topic test, Current leader 1's latest 
> offset 0 is less than replica 2's latest offset 151 
> (kafka.server.ReplicaFetcherThread)
> I am able to recover by setting unclean.leader.election.enable=true on my 
> brokers.
> I'm trying to understand a couple things:
> * In step 10, why is broker1 allowed to resume leadership even though it has 
> no data?
> * In step 10, why is it necessary to stop the entire broker due to one 
> partition that is in this state? Wouldn't it be possible for the broker to 
> continue to serve traffic for all the other topics, and just mark this one as 
> unavailable?
> * Would it make sense to allow an operator to manually specify which broker 
> they want to become the new master? This would give me more control over how 
> much data loss I am willing to handle. In this case, I would want broker2 to 
> become the new master. Or, is that possible and I just don't know how to do 
> it?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-4106) Consumer / add configure method to PartitionAssignor interface

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15453098#comment-15453098
 ] 

Guozhang Wang edited comment on KAFKA-4106 at 8/31/16 7:11 PM:
---

[~fhussonnois] {{PartitionAssignor}} extends {{Configurable}} and its 
{{configure}} function is auto triggered when the object is created.


was (Author: guozhang):
[~fhussonnois] {PartitionAssignor} extends {Configurable} and its {configure} 
function is auto triggered when the object is created.

> Consumer / add configure method to PartitionAssignor interface
> --
>
> Key: KAFKA-4106
> URL: https://issues.apache.org/jira/browse/KAFKA-4106
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients, consumer
>Affects Versions: 0.10.0.1
>Reporter: Florian Hussonnois
>Priority: Minor
>
> Currently, we can implement a custom PartitionAssignor which will forward 
> user data that will be used during the assignments protocol. For example, 
> data can be used to implement a rack-aware assignor
> However, currently we cannot dynamically configure a PartitionAssignor 
> instance.
> It would be nice to add a method configure(Map PartitionAssignor interface. Then, this method will be invoked by the 
> KafkaConsumer  on each assignor, as this is do for deserializers.
> The code modifications are pretty straight-forward but involve modifying the 
> public interface PartitionAssignor. Does that mean this JIRA needs a KIP ?
> I can contribute to that improvement.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3144) report members with no assigned partitions in ConsumerGroupCommand

[Vahid Hashemian (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15453096#comment-15453096
 ] 

Vahid Hashemian commented on KAFKA-3144:


[~junrao] [~alexlod] Thanks for the feedback. I'll look into adding these to 
the current PR.

> report members with no assigned partitions in ConsumerGroupCommand
> --
>
> Key: KAFKA-3144
> URL: https://issues.apache.org/jira/browse/KAFKA-3144
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: Vahid Hashemian
>  Labels: newbie
>
> A couple of suggestions on improving ConsumerGroupCommand. 
> 1. It would be useful to list members with no assigned partitions when doing 
> describe in ConsumerGroupCommand.
> 2. Currently, we show the client.id of each member when doing describe in 
> ConsumerGroupCommand. Since client.id is supposed to be the logical 
> application id, all members in the same group are supposed to set the same 
> client.id. So, it would be clearer if we show the client id as well as the 
> member id.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Kafka consumers unable to process message

[Ghosh, Achintya (Contractor)]

Hi there,

Kafka consumer gets stuck at consumer.poll() method if my current datacenter is 
down and replicated messages are in remote datacenter.

How to solve that issue?

Thanks
Achintya

[GitHub] kafka pull request #1805: MINOR: onCompletion: metadata valid only if no exc...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1805


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [VOTE] KIP-33 - Add a time based log index

[Guozhang Wang]

Some of the streams integration tests also encounters this issue where the
timestamps we are using in the test are very small (e.g. 1,2,3...) which
cause the log to roll upon each append, and the old segment gets deleted
very soon. Arguably this can be resolved to enforce LogAppendTime
configuration on the embedded server.

+1 on the proposed change, makes sense to me.

Guozhang


On Tue, Aug 30, 2016 at 4:33 PM, Becket Qin  wrote:

> Hi folks,
>
> Here is another update on the change of time based log rolling.
>
> After the latest implementation, we encountered KAFKA-4099. The issue is
> that if users move replicas, for the messages in the old segments, the new
> replica will create one log segment for each message. The root cause of
> this is we are comparing the wall clock time with the message timestamp. A
> solution to that is also described in KAFKA-4099, which is to change the
> log rolling purely based on the timestamp in the messages. More
> specifically, we roll out the log segment if the timestamp in the current
> message is greater than the timestamp of the first message in the segment
> by more than log.roll.ms. This approach is wall clock independent and
> should solve the problem. With message.timestamp.difference.max.ms
> configuration, we can achieve 1) the log segment will be rolled out in a
> bounded time, 2) no excessively large timestamp will be accepted and cause
> frequent log rolling.
>
> Any concern regarding this change?
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Mon, Jun 13, 2016 at 2:30 PM, Guozhang Wang  wrote:
>
> > Thanks Jiangjie,
> >
> > I see the need for sensitive data purging, the above proposed change
> LGTM.
> > One minor concern is that a wrongly marked timestamp on the first record
> > could cause the segment to roll much later / earlier, though it may be
> > rare.
> >
> > Guozhang
> >
> > On Fri, Jun 10, 2016 at 10:07 AM, Becket Qin 
> wrote:
> >
> > > Hi,
> > >
> > > During the implementation of KIP-33, we found it might be useful to
> have
> > a
> > > more deterministic time based log rolling than what proposed in the
> KIP.
> > >
> > > The current KIP proposal uses the largest timestamp in the segment for
> > time
> > > based rolling. The active log segment only rolls when there is no
> message
> > > appended in max.roll.ms since the largest timestamp in the segment.
> i.e.
> > > the rolling time may change if user keeping appending messages into the
> > > segment. This may not be a desirable behavior for people who have
> > sensitive
> > > data and want to make sure they are removed after some time.
> > >
> > > To solve the above issue, we want to modify the KIP proposal regarding
> > the
> > > time based rolling to the following behavior. The time based log
> rolling
> > > will be based on the first message with a timestamp in the log segment
> if
> > > there is such a message. If no message in the segment has a timestamp,
> > the
> > > time based log rolling will still be based on log segment create time,
> > > which is the same as we are doing now. The reasons we don't want to
> > always
> > > roll based on file create time are because 1) the message timestamp may
> > be
> > > assigned by clients which can be different from the create time of the
> > log
> > > segment file. 2) On some Linux, the file create time is not available,
> so
> > > using segment file create time may not always work.
> > >
> > > Do people have any concern for this change? I will update the KIP if
> > people
> > > think the change is OK.
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > >
> > > On Tue, Apr 19, 2016 at 6:27 PM, Becket Qin 
> > wrote:
> > >
> > > > Thanks Joel and Ismael. I just updated the KIP based on your
> feedback.
> > > >
> > > > KIP-33 has passed with +4 (binding) and +2 (non-binding)
> > > >
> > > > Thanks everyone for the reading, feedback and voting!
> > > >
> > > > Jiangjie (Becket) Qin
> > > >
> > > > On Tue, Apr 19, 2016 at 5:25 PM, Ismael Juma 
> > wrote:
> > > >
> > > >> Thanks Becket. I think it would be nice to update the KIP with
> regards
> > > to
> > > >> point 3 and 4.
> > > >>
> > > >> In any case, +1 (non-binding)
> > > >>
> > > >> Ismael
> > > >>
> > > >> On Tue, Apr 19, 2016 at 2:03 AM, Becket Qin 
> > > wrote:
> > > >>
> > > >> > Thanks for the comments Ismael. Please see the replies inline.
> > > >> >
> > > >> > On Mon, Apr 18, 2016 at 6:50 AM, Ismael Juma 
> > > wrote:
> > > >> >
> > > >> > > Hi Jiangjie,
> > > >> > >
> > > >> > > Thanks for the KIP, it's a nice improvement. Since it seems like
> > we
> > > >> have
> > > >> > > been using the voting thread for discussion, I'll do the same.
> > > >> > >
> > > >> > > A few minor comments/questions:
> > > >> > >
> > > >> > > 1. The proposed name for the time index file
> > > >> > `SegmentBaseOffset.timeindex`.
> > > >> > > Would 

Re: Question regarding Producer and Duplicates

[Guozhang Wang]

Hi Florian,

The broker will accept a batch of records as a whole or reject them as a
whole unless it encounters an IOException while trying to append the
messages, which will be treated as a fatal error anyways.

Duplicates usually happen when the whole batch is accepted but the ack was
not delivered in time, and hence it was re-tried.


Guozhang


On Tue, Aug 30, 2016 at 2:45 AM, Florian Hussonnois 
wrote:

> Hi all,
>
> I am using kafka_2.11-0.10.0.1, my understanding is that the producer API
> batches records per partition to send efficient requests. We can configure
> batch.size to increase the throughtput.
>
> However, in case of failure all records within the batch failed ? If that
> is true,  does that mean that increasing batch.size can also increase the
> number of duplicates in case of retries ?
>
> Thanks,
>
> Florian.
>



-- 
-- Guozhang

Re: Kafka KIP meeting Aug 30 at 11:00am PST

[Ismael Juma]

OK, thanks. Just making sure. :)

Ismael

On 31 Aug 2016 5:47 pm, "Jim Jagielski"  wrote:

> As I said, just a general reminder...
>
> It is good, I think, to have them every so often to reinforce things.
> Sometimes it is easy to get "lazy" about it :)
>
> > On Aug 31, 2016, at 12:34 PM, Ismael Juma  wrote:
> >
> > Hi Jim,
> >
> > Thanks for the feedback. More inline.
> >
> > On Wed, Aug 31, 2016 at 5:19 PM, Jim Jagielski  wrote:
> >
> >> Just a reminder that these kinds of meetings are, by their very nature,
> >> synchronous and disenfranchise those members of the community unable to
> >> attend.
> >>
> >> It is great that discussions and videos are brought back to the
> >> list BUT they should not be "this is what has been decided" or
> >> "this is what we are going to do" kinds of results.
> >>
> >
> > Was there something specific of this sort that concerned you? In general,
> > we follow up on the mailing list (and wiki for KIPs). Also, the mailing
> > list is where all the votes take place. I couldn't find any examples
> where
> > we decided a course of action in the meeting, but I could have missed
> > something.
> >
> > Please recall the Apache theme: If it didn't happen on the (dev) list,
> >> it didn't happen at all.
> >>
> >
> > Of course.
> >
> > Ismael
>
>

[jira] [Commented] (KAFKA-4106) Consumer / add configure method to PartitionAssignor interface

[Jason Gustafson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15452913#comment-15452913
 ] 

Jason Gustafson commented on KAFKA-4106:


[~fhussonnois] I think you might be able to do this already if you have your 
{{PartitionAssignor}} implement {{org.apache.kafka.common.Configurable}}.

> Consumer / add configure method to PartitionAssignor interface
> --
>
> Key: KAFKA-4106
> URL: https://issues.apache.org/jira/browse/KAFKA-4106
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients, consumer
>Affects Versions: 0.10.0.1
>Reporter: Florian Hussonnois
>Priority: Minor
>
> Currently, we can implement a custom PartitionAssignor which will forward 
> user data that will be used during the assignments protocol. For example, 
> data can be used to implement a rack-aware assignor
> However, currently we cannot dynamically configure a PartitionAssignor 
> instance.
> It would be nice to add a method configure(Map PartitionAssignor interface. Then, this method will be invoked by the 
> KafkaConsumer  on each assignor, as this is do for deserializers.
> The code modifications are pretty straight-forward but involve modifying the 
> public interface PartitionAssignor. Does that mean this JIRA needs a KIP ?
> I can contribute to that improvement.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4103) DumpLogSegments cannot print data from offsets topic

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15452869#comment-15452869
 ] 

ASF GitHub Bot commented on KAFKA-4103:
---

GitHub user hachikuji opened a pull request:

https://github.com/apache/kafka/pull/1807

KAFKA-4103: Fix regression in DumpLogSegments offsets decoder



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hachikuji/kafka KAFKA-4103

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1807.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1807


commit 1e80d26256d2a09db88601a387244c47a9f44e2c
Author: Jason Gustafson 
Date:   2016-08-31T17:47:12Z

KAFKA-4103: Fix regression in DumpLogSegments offsets decoder




> DumpLogSegments cannot print data from offsets topic
> 
>
> Key: KAFKA-4103
> URL: https://issues.apache.org/jira/browse/KAFKA-4103
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
>Reporter: Ewen Cheslack-Postava
>Assignee: Jason Gustafson
>Priority: Blocker
> Fix For: 0.10.1.0
>
>
> It looks like there's been a regression in the DumpLogSegments tool. I'm 
> marking it a blocker since it appears we can no longer dump offset 
> information from this tool, which makes it really hard to debug anything 
> related to __consumer_offsets.
> The 0.10.0 branch seems to work fine, but even with offsets log files 
> generated using only old formats (0.10.0 branch), the DumpLogSegments tool 
> from trunk (i.e. 0.10.1.0-SNAPSHOT with latest githash 
> b91eeac9438b8718c410045b0e9191296ebb536d as of reporting this) will cause the 
> exception below. This was found while doing some basic testing of KAFKA-4062.
> {quote}
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> offset: 0 position: 0 CreateTime: 1472615183913 isvalid: true payloadsize: 
> 199 magic: 1 compresscodec: NoCompressionCodec crc: 2036280914 keysize: 
> 26Exception in thread "main" java.util.IllegalFormatConversionException: x != 
> scala.math.BigInt
>   at 
> java.util.Formatter$FormatSpecifier.failConversion(Formatter.java:4045)
>   at 
> java.util.Formatter$FormatSpecifier.printInteger(Formatter.java:2748)
>   at 
> java.util.Formatter$FormatSpecifier.print(Formatter.java:2702)
>   at java.util.Formatter.format(Formatter.java:2488)
>   at java.util.Formatter.format(Formatter.java:2423)
>   at java.lang.String.format(String.java:2792)
>   at 
> kafka.tools.DumpLogSegments$OffsetsMessageParser.kafka$tools$DumpLogSegments$OffsetsMessageParser$$hex(DumpLogSegments.scala:240)
>   at 
> kafka.tools.DumpLogSegments$OffsetsMessageParser$$anonfun$3.apply(DumpLogSegments.scala:272)
>   at 
> kafka.tools.DumpLogSegments$OffsetsMessageParser$$anonfun$3.apply(DumpLogSegments.scala:262)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at scala.collection.immutable.List.foreach(List.scala:318)
>   at 
> scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
>   at 
> scala.collection.AbstractTraversable.map(Traversable.scala:105)
>   at 
> kafka.tools.DumpLogSegments$OffsetsMessageParser.parseGroupMetadata(DumpLogSegments.scala:262)
>   at 
> kafka.tools.DumpLogSegments$OffsetsMessageParser.parse(DumpLogSegments.scala:290)
>   at 
> kafka.tools.DumpLogSegments$$anonfun$kafka$tools$DumpLogSegments$$dumpLog$1$$anonfun$apply$3.apply(DumpLogSegments.scala:332)
>   at 
> kafka.tools.DumpLogSegments$$anonfun$kafka$tools$DumpLogSegments$$dumpLog$1$$anonfun$apply$3.apply(DumpLogSegments.scala:312)
>   at scala.collection.Iterator$class.foreach(Iterator.scala:727)
>   at 
> kafka.utils.IteratorTemplate.foreach(IteratorTemplate.scala:30)
>   at 
> kafka.tools.DumpLogSegments$$anonfun$kafka$tools$DumpLogSegments$$dumpLog$1.apply(DumpLogSegments.scala:312)
>   at 
> kafka.tools.DumpLogSegments$$anonfun$kafka$tools$DumpLogSegments$$dumpLog$1.apply(DumpLogSegments.scala:310)
>   at scala.collection.Iterator$class.foreach(Iterator.scala:727)
>   at 
> kafka.utils.IteratorTemplate.foreach(IteratorTemplate.scala:30)
>   at 
> kafka.tools.DumpLogSegments$.kafka$tools$DumpLogSegments$$dumpLog(DumpLogSegments.scala:310)
>   at 
> 

[GitHub] kafka pull request #1807: KAFKA-4103: Fix regression in DumpLogSegments offs...

[hachikuji]

GitHub user hachikuji opened a pull request:

https://github.com/apache/kafka/pull/1807

KAFKA-4103: Fix regression in DumpLogSegments offsets decoder



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hachikuji/kafka KAFKA-4103

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1807.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1807


commit 1e80d26256d2a09db88601a387244c47a9f44e2c
Author: Jason Gustafson 
Date:   2016-08-31T17:47:12Z

KAFKA-4103: Fix regression in DumpLogSegments offsets decoder




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Kafka KIP meeting Aug 30 at 11:00am PST

[Jim Jagielski]

As I said, just a general reminder...

It is good, I think, to have them every so often to reinforce things.
Sometimes it is easy to get "lazy" about it :)

> On Aug 31, 2016, at 12:34 PM, Ismael Juma  wrote:
> 
> Hi Jim,
> 
> Thanks for the feedback. More inline.
> 
> On Wed, Aug 31, 2016 at 5:19 PM, Jim Jagielski  wrote:
> 
>> Just a reminder that these kinds of meetings are, by their very nature,
>> synchronous and disenfranchise those members of the community unable to
>> attend.
>> 
>> It is great that discussions and videos are brought back to the
>> list BUT they should not be "this is what has been decided" or
>> "this is what we are going to do" kinds of results.
>> 
> 
> Was there something specific of this sort that concerned you? In general,
> we follow up on the mailing list (and wiki for KIPs). Also, the mailing
> list is where all the votes take place. I couldn't find any examples where
> we decided a course of action in the meeting, but I could have missed
> something.
> 
> Please recall the Apache theme: If it didn't happen on the (dev) list,
>> it didn't happen at all.
>> 
> 
> Of course.
> 
> Ismael

Re: Kafka KIP meeting Aug 30 at 11:00am PST

[Ismael Juma]

Hi Jim,

Thanks for the feedback. More inline.

On Wed, Aug 31, 2016 at 5:19 PM, Jim Jagielski  wrote:

> Just a reminder that these kinds of meetings are, by their very nature,
> synchronous and disenfranchise those members of the community unable to
> attend.
>
> It is great that discussions and videos are brought back to the
> list BUT they should not be "this is what has been decided" or
> "this is what we are going to do" kinds of results.
>

Was there something specific of this sort that concerned you? In general,
we follow up on the mailing list (and wiki for KIPs). Also, the mailing
list is where all the votes take place. I couldn't find any examples where
we decided a course of action in the meeting, but I could have missed
something.

Please recall the Apache theme: If it didn't happen on the (dev) list,
> it didn't happen at all.
>

Of course.

Ismael

[GitHub] kafka pull request #1483: KAFKA-3799: Enable SSL endpoint validation in syst...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1483


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3799) Turn on endpoint validation in SSL system tests

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15452666#comment-15452666
 ] 

ASF GitHub Bot commented on KAFKA-3799:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1483


> Turn on endpoint validation in SSL system tests
> ---
>
> Key: KAFKA-3799
> URL: https://issues.apache.org/jira/browse/KAFKA-3799
> Project: Kafka
>  Issue Type: Test
>  Components: system tests
>Affects Versions: 0.10.0.0
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
> Fix For: 0.10.1.0
>
>
> Endpoint validation is off by default and currently system tests are run 
> without endpoint validation. It will be better to run system tests with 
> endpoint validation turned on. KAFKA-3665 will be enabling validation by 
> default as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] KIP-63: Unify store and downstream caching in streams

[Jim Jagielski]

+1
> On Aug 25, 2016, at 6:57 AM, Eno Thereska  wrote:
> 
> Hi folks,
> 
> We'd like to start the vote for KIP-63. At this point the Wiki addresses
> all previous questions and we believe the PoC is feature-complete.
> 
> Thanks
> Eno

[jira] [Updated] (KAFKA-3799) Turn on endpoint validation in SSL system tests

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-3799:
-
   Resolution: Fixed
Fix Version/s: 0.10.1.0
   Status: Resolved  (was: Patch Available)

Issue resolved by pull request 1483
[https://github.com/apache/kafka/pull/1483]

> Turn on endpoint validation in SSL system tests
> ---
>
> Key: KAFKA-3799
> URL: https://issues.apache.org/jira/browse/KAFKA-3799
> Project: Kafka
>  Issue Type: Test
>  Components: system tests
>Affects Versions: 0.10.0.0
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
> Fix For: 0.10.1.0
>
>
> Endpoint validation is off by default and currently system tests are run 
> without endpoint validation. It will be better to run system tests with 
> endpoint validation turned on. KAFKA-3665 will be enabling validation by 
> default as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Kafka KIP meeting Aug 30 at 11:00am PST

[Jim Jagielski]

Just a reminder that these kinds of meetings are, by their very nature,
synchronous and disenfranchise those members of the community unable to
attend.

It is great that discussions and videos are brought back to the
list BUT they should not be "this is what has been decided" or
"this is what we are going to do" kinds of results.

Please recall the Apache theme: If it didn't happen on the (dev) list,
it didn't happen at all.

> On Aug 30, 2016, at 3:01 PM, Jun Rao  wrote:
> 
> The following are the notes from today's KIP discussion.
> 
>   - KIP48 (delegation tokens): Harsha will update the wiki with more
>   details on how to use delegation tokens and how to configure it.
>   - KIP-78 (cluster id): There was discussion on adding human readable
>   tags later. No major concerns.
> 
> The video will be uploaded soon in https://cwiki.apache.org/co
> nfluence/display/KAFKA/Kafka+Improvement+Proposals .
> 
> Thanks,
> 
> 
> On Fri, Aug 26, 2016 at 10:40 AM, Jun Rao  wrote:
> 
>> Hi, Everyone.,
>> 
>> We plan to have a Kafka KIP meeting this coming Tuesday at 11:00am PST. If
>> you plan to attend but haven't received an invite, please let me know.
>> The following is the tentative agenda.
>> 
>> Agenda:
>> KIP-48: delegation tokens
>> 
>> Thanks,
>> 
>> Jun
>> 

Re: [VOTE] KIP-77: Improve Kafka Streams Join Semantics

[Jim Jagielski]

+1

> On Aug 29, 2016, at 5:50 AM, Matthias J. Sax  wrote:
> 
> I’d like to initiate the voting process for KIP-77:
> 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-77%3A+Improve+Kafka+Streams+Join+Semantics
> 
> -Matthias
> 
> 

Re: [VOTE] KIP-63: Unify store and downstream caching in streams

[Michael Noll]

+1 (non-binding)

On Fri, Aug 26, 2016 at 3:10 PM, Bill Bejeck  wrote:

> +1
>
> On Thu, Aug 25, 2016 at 11:45 AM, Matthias J. Sax 
> wrote:
>
> > +1
> >
> > On 08/25/2016 04:22 PM, Damian Guy wrote:
> > > +1
> > >
> > > On Thu, 25 Aug 2016 at 11:57 Eno Thereska 
> > wrote:
> > >
> > >> Hi folks,
> > >>
> > >> We'd like to start the vote for KIP-63. At this point the Wiki
> addresses
> > >> all previous questions and we believe the PoC is feature-complete.
> > >>
> > >> Thanks
> > >> Eno
> > >>
> > >
> >
> >
>



-- 
*Michael G. Noll*
Product Manager | Confluent
+1 650 453 5860 | @miguno 
Follow us: Twitter  | Blog

Re: [VOTE] KIP-77: Improve Kafka Streams Join Semantics

[Michael Noll]

+1 (non-binding)

On Wed, Aug 31, 2016 at 6:18 AM, Neha Narkhede  wrote:

> +1 (binding)
>
> On Tue, Aug 30, 2016 at 5:33 PM Ewen Cheslack-Postava 
> wrote:
>
> > +1 (binding)
> >
> > I think the major gap I notice in the PR is a lack of docs updates to
> > notify people of the change. Given these are improvements and streams is
> > still new, I wouldn't necessarily call them out as anything critical, but
> > the changes should be noted somewhere.
> >
> > -Ewen
> >
> > On Tue, Aug 30, 2016 at 3:53 PM, Guozhang Wang 
> wrote:
> >
> > > +1 (binding)
> > >
> > > Thanks!
> > >
> > > On Tue, Aug 30, 2016 at 2:42 AM, Damian Guy 
> > wrote:
> > >
> > > > +1
> > > >
> > > > On Mon, 29 Aug 2016 at 18:07 Eno Thereska 
> > > wrote:
> > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > > On 29 Aug 2016, at 12:22, Bill Bejeck  wrote:
> > > > > >
> > > > > > +1
> > > > > >
> > > > > > On Mon, Aug 29, 2016 at 5:50 AM, Matthias J. Sax <
> > > > matth...@confluent.io>
> > > > > > wrote:
> > > > > >
> > > > > >> I’d like to initiate the voting process for KIP-77:
> > > > > >>
> > > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > >> 77%3A+Improve+Kafka+Streams+Join+Semantics
> > > > > >>
> > > > > >> -Matthias
> > > > > >>
> > > > > >>
> > > > > >>
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
> >
> >
> > --
> > Thanks,
> > Ewen
> >
> --
> Thanks,
> Neha
>



-- 
*Michael G. Noll*
Product Manager | Confluent
+1 650 453 5860 | @miguno 
Follow us: Twitter  | Blog

[GitHub] kafka pull request #1806: KAFKA-4105: Queryable state tests

[enothereska]

GitHub user enothereska opened a pull request:

https://github.com/apache/kafka/pull/1806

KAFKA-4105: Queryable state tests



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/enothereska/kafka queryable-state-tests

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1806.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1806


commit b8f2bac04290a75e053e3186e3ab5b449a0bb79e
Author: Eno Thereska 
Date:   2016-08-30T13:25:10Z

Concurrency integration test

commit 1b68275c17b34dc61646bf670b28ce599986df57
Author: Eno Thereska 
Date:   2016-08-30T17:34:08Z

First pass at query on rebalance

commit 68cb493979b750d71af4a4cd7944858da2ef1728
Author: Eno Thereska 
Date:   2016-08-31T10:18:02Z

Checkpoint

commit ee1e5cefa52ff281d6e54c90e17537c5e854cb5c
Author: Eno Thereska 
Date:   2016-08-31T11:26:28Z

Checkpoint

commit 20cd7cebf173d585aeef9e8c218b6d91bcab6c0c
Author: Eno Thereska 
Date:   2016-08-31T15:46:39Z

Added check when thread shut down




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4105) Queryable state tests for concurrency and rebalancing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15452590#comment-15452590
 ] 

ASF GitHub Bot commented on KAFKA-4105:
---

GitHub user enothereska opened a pull request:

https://github.com/apache/kafka/pull/1806

KAFKA-4105: Queryable state tests



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/enothereska/kafka queryable-state-tests

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1806.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1806


commit b8f2bac04290a75e053e3186e3ab5b449a0bb79e
Author: Eno Thereska 
Date:   2016-08-30T13:25:10Z

Concurrency integration test

commit 1b68275c17b34dc61646bf670b28ce599986df57
Author: Eno Thereska 
Date:   2016-08-30T17:34:08Z

First pass at query on rebalance

commit 68cb493979b750d71af4a4cd7944858da2ef1728
Author: Eno Thereska 
Date:   2016-08-31T10:18:02Z

Checkpoint

commit ee1e5cefa52ff281d6e54c90e17537c5e854cb5c
Author: Eno Thereska 
Date:   2016-08-31T11:26:28Z

Checkpoint

commit 20cd7cebf173d585aeef9e8c218b6d91bcab6c0c
Author: Eno Thereska 
Date:   2016-08-31T15:46:39Z

Added check when thread shut down




> Queryable state tests for concurrency and rebalancing
> -
>
> Key: KAFKA-4105
> URL: https://issues.apache.org/jira/browse/KAFKA-4105
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.1.0
>Reporter: Eno Thereska
>Assignee: Eno Thereska
> Fix For: 0.10.1.0
>
>
> The queryable state feature (KIP-67) needs more tests on concurrent queries 
> and queries during rebalancing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-78: Cluster Id

[Jun Rao]

Ismael,

Thanks for the proposal. It looks good overall. Just a comment below.

We are adding the following interface in ClusterListener and Cluster
includes all brokers' endpoint and the metadata of topic partitions.

void onClusterUpdate(Cluster cluster);


On the broker side, will that method be called when there is broker or
topic/partition change in metadata cache? Another thing is that Cluster
only includes one endpoint. This makes sense for the client. However, on
the broker side, it's not clear which endpoint should be used.

In general, I am not sure how useful it is for serializers, interceptors,
metric reporters to know all brokers endpoints and topic/partition metadata.

I was thinking we could instead pass in sth like a ClusterResourceMeta and
just include the clusterId for now. This way, we can guarantee that
onClusterUpdate()
will only be called once and it's easier to implement on the broker side.

Jun


On Wed, Aug 31, 2016 at 1:24 AM, Ismael Juma  wrote:

> Thanks for the feedback Guozhang. Comment inline.
>
> On Wed, Aug 31, 2016 at 2:49 AM, Guozhang Wang  wrote:
>
> > About logging / debugging with the cluster id: I think the random UUID
> > itself may not be very helpful for human-readable debugging information,
> > and we'd better use the cluster name mentioned in future work in logging.
> >
>
> We can also add the human-readable value once it's available. However, the
> random UUID is still useful now. After all, we use Git commit hashes in
> many places and they are significantly longer than what we are proposing
> here (40 instead of 22 characters) . When comparing by eye, one can often
> just look at the first few characters to distinguish. Does that make sense?
>
> Ismael
>

[jira] [Created] (KAFKA-4106) Consumer / add configure method to PartitionAssignor interface

[Florian Hussonnois (JIRA)]

Florian Hussonnois created KAFKA-4106:
-

 Summary: Consumer / add configure method to PartitionAssignor 
interface
 Key: KAFKA-4106
 URL: https://issues.apache.org/jira/browse/KAFKA-4106
 Project: Kafka
  Issue Type: Improvement
  Components: clients, consumer
Affects Versions: 0.10.0.1
Reporter: Florian Hussonnois
Priority: Minor


Currently, we can implement a custom PartitionAssignor which will forward user 
data that will be used during the assignments protocol. For example, data can 
be used to implement a rack-aware assignor

However, currently we cannot dynamically configure a PartitionAssignor instance.

It would be nice to add a method configure(Map

[jira] [Updated] (KAFKA-4104) Queryable state metadata is sometimes invalid

[Damian Guy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Damian Guy updated KAFKA-4104:
--
Status: Patch Available  (was: In Progress)

> Queryable state metadata is sometimes invalid
> -
>
> Key: KAFKA-4104
> URL: https://issues.apache.org/jira/browse/KAFKA-4104
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.1.0
>Reporter: Eno Thereska
>Assignee: Damian Guy
> Fix For: 0.10.1.0
>
>
> The streams.metadataForKey method sometimes fails because the cluster-wide 
> metadata is invalid/null in non-leader nodes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1805: MINOR: onCompletion: metadata valid only if no exc...

[ybyzek]

GitHub user ybyzek opened a pull request:

https://github.com/apache/kafka/pull/1805

MINOR: onCompletion: metadata valid only if no exception

Modifies example in doc change

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ybyzek/kafka onComplete_doc

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1805.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1805


commit 0bd8796063677349da4ee34f4bb3f24bcde193c1
Author: ybyzek 
Date:   2016-08-31T13:51:19Z

onCompletion: metadata valid only if no exception




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Work started] (KAFKA-4069) Forward records in context of cache flushing/eviction

[Damian Guy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-4069 started by Damian Guy.
-
> Forward records in context of cache flushing/eviction
> -
>
> Key: KAFKA-4069
> URL: https://issues.apache.org/jira/browse/KAFKA-4069
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Affects Versions: 0.10.1.0
>Reporter: Eno Thereska
>Assignee: Damian Guy
> Fix For: 0.10.1.0
>
>
> When the cache is in place, records should we forwarded downstream when they 
> are evicted or flushed from the cache. 
> This is a major structural change to the internals of the code, moving from 
> having a single record outstanding inside a task to potentially having 
> several records outstanding. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3595) Add capability to specify replication compact option for stream store

[Damian Guy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Damian Guy updated KAFKA-3595:
--
Status: Patch Available  (was: In Progress)

> Add capability to specify replication compact option for stream store
> -
>
> Key: KAFKA-3595
> URL: https://issues.apache.org/jira/browse/KAFKA-3595
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.1.0
>Reporter: Henry Cai
>Assignee: Damian Guy
>Priority: Minor
>  Labels: user-experience
> Fix For: 0.10.1.0
>
>
> Currently state store replication always go through a compact kafka topic. 
> For some state stores, e.g. JoinWindow, there are no duplicates in the store, 
> there is not much benefit using a compacted topic.
> The problem of using compacted topic is the records can stay in kafka broker 
> forever. In my use case, my key is ad_id, it's incrementing all the time, not 
> bounded, I am worried the disk space on broker for that topic will go forever.
> I think we either need the capability to purge the compacted records on 
> broker, or allow us to specify different compact option for state store 
> replication.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Work started] (KAFKA-3595) Add capability to specify replication compact option for stream store

[Damian Guy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-3595 started by Damian Guy.
-
> Add capability to specify replication compact option for stream store
> -
>
> Key: KAFKA-3595
> URL: https://issues.apache.org/jira/browse/KAFKA-3595
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.1.0
>Reporter: Henry Cai
>Assignee: Damian Guy
>Priority: Minor
>  Labels: user-experience
> Fix For: 0.10.1.0
>
>
> Currently state store replication always go through a compact kafka topic. 
> For some state stores, e.g. JoinWindow, there are no duplicates in the store, 
> there is not much benefit using a compacted topic.
> The problem of using compacted topic is the records can stay in kafka broker 
> forever. In my use case, my key is ad_id, it's incrementing all the time, not 
> bounded, I am worried the disk space on broker for that topic will go forever.
> I think we either need the capability to purge the compacted records on 
> broker, or allow us to specify different compact option for state store 
> replication.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Work started] (KAFKA-4104) Queryable state metadata is sometimes invalid

[Damian Guy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-4104 started by Damian Guy.
-
> Queryable state metadata is sometimes invalid
> -
>
> Key: KAFKA-4104
> URL: https://issues.apache.org/jira/browse/KAFKA-4104
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.1.0
>Reporter: Eno Thereska
>Assignee: Damian Guy
> Fix For: 0.10.1.0
>
>
> The streams.metadataForKey method sometimes fails because the cluster-wide 
> metadata is invalid/null in non-leader nodes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4104) Queryable state metadata is sometimes invalid

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15452031#comment-15452031
 ] 

ASF GitHub Bot commented on KAFKA-4104:
---

GitHub user dguy opened a pull request:

https://github.com/apache/kafka/pull/1804

KAFKA-4104: Queryable state metadata is sometimes invalid

If the thread or process is not the coordinator the Cluster instance in 
StreamPartitionAssignor will always be null. This builds an instance of the 
Cluster with the metadata associated with the Assignment

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dguy/kafka kafka-4104

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1804.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1804


commit 6cc1149d22ef85d6b0c5ace3d7e624f4f66fefa6
Author: Damian Guy 
Date:   2016-08-31T11:51:16Z

build Cluster in StreamPartitionAssignor.onAssignment




> Queryable state metadata is sometimes invalid
> -
>
> Key: KAFKA-4104
> URL: https://issues.apache.org/jira/browse/KAFKA-4104
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.1.0
>Reporter: Eno Thereska
>Assignee: Damian Guy
> Fix For: 0.10.1.0
>
>
> The streams.metadataForKey method sometimes fails because the cluster-wide 
> metadata is invalid/null in non-leader nodes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1804: KAFKA-4104: Queryable state metadata is sometimes ...

[dguy]

GitHub user dguy opened a pull request:

https://github.com/apache/kafka/pull/1804

KAFKA-4104: Queryable state metadata is sometimes invalid

If the thread or process is not the coordinator the Cluster instance in 
StreamPartitionAssignor will always be null. This builds an instance of the 
Cluster with the metadata associated with the Assignment

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dguy/kafka kafka-4104

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1804.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1804


commit 6cc1149d22ef85d6b0c5ace3d7e624f4f66fefa6
Author: Damian Guy 
Date:   2016-08-31T11:51:16Z

build Cluster in StreamPartitionAssignor.onAssignment




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-4104) Queryable state metadata is sometimes invalid

[Eno Thereska (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eno Thereska updated KAFKA-4104:

Assignee: Damian Guy

> Queryable state metadata is sometimes invalid
> -
>
> Key: KAFKA-4104
> URL: https://issues.apache.org/jira/browse/KAFKA-4104
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.1.0
>Reporter: Eno Thereska
>Assignee: Damian Guy
> Fix For: 0.10.1.0
>
>
> The streams.metadataForKey method sometimes fails because the cluster-wide 
> metadata is invalid/null in non-leader nodes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-4105) Queryable state tests for concurrency and rebalancing

[Eno Thereska (JIRA)]

Eno Thereska created KAFKA-4105:
---

 Summary: Queryable state tests for concurrency and rebalancing
 Key: KAFKA-4105
 URL: https://issues.apache.org/jira/browse/KAFKA-4105
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 0.10.1.0
Reporter: Eno Thereska
Assignee: Eno Thereska
 Fix For: 0.10.1.0


The queryable state feature (KIP-67) needs more tests on concurrent queries and 
queries during rebalancing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-4104) Queryable state metadata is sometimes invalid

[Eno Thereska (JIRA)]

Eno Thereska created KAFKA-4104:
---

 Summary: Queryable state metadata is sometimes invalid
 Key: KAFKA-4104
 URL: https://issues.apache.org/jira/browse/KAFKA-4104
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 0.10.1.0
Reporter: Eno Thereska
 Fix For: 0.10.1.0


The streams.metadataForKey method sometimes fails because the cluster-wide 
metadata is invalid/null in non-leader nodes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Newbie JIRAs

[Matthias J. Sax]

Hi and welcome to Kafka community!

You can browse the Jira board for tickets with labels "newbie" or
"beginner":

Like this:
https://issues.apache.org/jira/browse/KAFKA-3827?jql=project%20%3D%20KAFKA%20AND%20status%20in%20%28Open%2C%20Reopened%29%20AND%20labels%20in%20%28newbie%2C%20%22newbie%2B%2B%22%2C%20newbiee%2C%20newbiew%2C%20newdev%2C%20beginner%2C%20beginners%29

-Matthias

On 08/31/2016 08:36 AM, Srabasti Banerjee wrote:
> Hi,I want to contribute to Apache Kafka.Can you please let me know any newbie 
> JIRAs I could pick up?ThanksSrabasti
> 



signature.asc
Description: OpenPGP digital signature

[jira] [Commented] (KAFKA-3587) LogCleaner fails due to incorrect offset map computation on a replica

[Zaiming Shi (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15451637#comment-15451637
 ] 

Zaiming Shi commented on KAFKA-3587:


any plan to port the fix back to 0.9 ?

> LogCleaner fails due to incorrect offset map computation on a replica
> -
>
> Key: KAFKA-3587
> URL: https://issues.apache.org/jira/browse/KAFKA-3587
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.1
> Environment: Linux
>Reporter: Kiran Pillarisetty
>Assignee: Edoardo Comar
> Fix For: 0.10.0.0
>
> Attachments: 0001-POC-improving-deduping-segments.patch
>
>
> Log Cleaner fails to compact a segment even when the number of messages in it 
> is less than the offset map.
> In version 0.9.0.1, (LogCleaner.scala -> buildOffsetMap()), LogCleaner 
> computes segment size by subtracting segment's base offset from the latest 
> offset ("segmentSize = segment.nextOffset() - segment.baseOffset").  This 
> works fine until you create another replica. When you create a replica, it's 
> segment could contain data which is already compacted on other brokers. 
> Depending up on the type of data, offset difference could be too big, larger 
> than the offset map (maxDesiredMapSize), and that causes LogCleaner to fail 
> on that segment.
> Scenario:
> - Kafka 0.9.0.1
> - Cluster has two brokers.
> - Server.properties:
> log.cleaner.enable=true
> log.cleaner.dedupe.buffer.size=10485760 #10MB
> log.roll.ms=30
> delete.topic.enable=true
> log.cleanup.policy=compact
> Steps to reproduce:
> 1. Create a topic with replication-factor of 1.
> ./kafka-topics.sh --zookeeper=localhost:2181 --create --topic 
> test.log.compact.1M --partitions 1 --replication-factor 1 --config 
> cleanup.policy=compact --config segment.ms=30
> 2. Use kafka-console-producer.sh to produce a single message with the 
> following key:
> LC1,{"test": "xyz"}
> 3. Use  kafka-console-producer.sh to produce a large number of messages with 
> the following key:
> LC2,{"test": "abc"}
> 4. Let log cleaner run. Make sure log is compacted.  Verify with:
>  ./kafka-run-class.sh kafka.tools.DumpLogSegments  --files 
> .log  --print-data-log
> Dumping .log
> Starting offset: 0
> offset: 0 position: 0 isvalid: true payloadsize: 11 magic: 0 compresscodec: 
> NoCompressionCodec crc: 3067045277 keysize: 11 key: LC1 payload: {"test": 
> "xyz"}
> offset: 7869818 position: 48 isvalid: true payloadsize: 11 magic: 0 
> compresscodec: NoCompressionCodec crc: 2668089711 keysize: 11 key: LC2 
> payload: {"test": "abc"}
> 5.  Increase Replication Factor to 2.  Followed these steps: 
> http://kafka.apache.org/documentation.html#basic_ops_increase_replication_factor
> 6. Notice that log cleaner fails to compact the newly created replica with 
> the following error.
> [2016-04-18 14:49:45,599] ERROR [kafka-log-cleaner-thread-0], Error due to  
> (kafka.log.LogCleaner)
> java.lang.IllegalArgumentException: requirement failed: 7206179 messages in 
> segment test.log.compact.1M-0/.log but offset map can fit 
> only 393215. You can increase log.cleaner.dedupe.buffer.size or decrease 
> log.cleaner.threads
> at scala.Predef$.require(Predef.scala:219)
> at 
> kafka.log.Cleaner$$anonfun$buildOffsetMap$4.apply(LogCleaner.scala:584)
> at 
> kafka.log.Cleaner$$anonfun$buildOffsetMap$4.apply(LogCleaner.scala:580)
> at 
> scala.collection.immutable.Stream$StreamWithFilter.foreach(Stream.scala:570)
> at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:580)
> at kafka.log.Cleaner.clean(LogCleaner.scala:322)
> at 
> kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:230)
> at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:208)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
> [2016-04-18 14:49:45,601] INFO [kafka-log-cleaner-thread-0], Stopped  
> (kafka.log.LogCleaner)
> 7. Examine the entries in the replica segment:
> ./kafka-run-class.sh kafka.tools.DumpLogSegments --files 
> .log  --print-data-log
> There are only 218418 messages in that segment.
> However, Log Cleaner seems to think that there are 7206179 messages in that 
> segment (as per the above error)
> Error stems from this line in LogCleaner.scala:
> """val segmentSize = segment.nextOffset() - segment.baseOffset"""
> In Replica's log segment file ( .log), ending offset is 
> 7206178. Beginning offset is 0.  That makes Log Cleaner think that there are 
> 7206179 messages in that segment although there are only 218418 messages in 
> it.
> IMO,  to address this kind of scenario, LogCleaner.scala should check for the 
> number of messages in the segment, instead of 

[jira] [Closed] (KAFKA-4097) "Server not found in kerberos database" issue while starting a Kafka server in a secured mode

[Syam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Syam closed KAFKA-4097.
---

> "Server not found in kerberos database" issue while starting a Kafka server 
> in a secured mode
> -
>
> Key: KAFKA-4097
> URL: https://issues.apache.org/jira/browse/KAFKA-4097
> Project: Kafka
>  Issue Type: Test
>Affects Versions: 0.10.0.1
>Reporter: Syam
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-78: Cluster Id

[Ismael Juma]

Thanks for the feedback Guozhang. Comment inline.

On Wed, Aug 31, 2016 at 2:49 AM, Guozhang Wang  wrote:

> About logging / debugging with the cluster id: I think the random UUID
> itself may not be very helpful for human-readable debugging information,
> and we'd better use the cluster name mentioned in future work in logging.
>

We can also add the human-readable value once it's available. However, the
random UUID is still useful now. After all, we use Git commit hashes in
many places and they are significantly longer than what we are proposing
here (40 instead of 22 characters) . When comparing by eye, one can often
just look at the first few characters to distinguish. Does that make sense?

Ismael

Newbie JIRAs

[Srabasti Banerjee]

Hi,I want to contribute to Apache Kafka.Can you please let me know any newbie 
JIRAs I could pick up?ThanksSrabasti