Re: kafka_2.11-0.9.0.1 crash with java coredump

I would suggest creating a JIRA and describing in detail what was going on in the cluster when this happened, and posting the associated broker / state change / controller logs. Thanks, Apurva On Wed, Dec 14, 2016 at 3:28 AM, Mazhar Shaikh wrote: > Hi All, > > I am

[jira] [Commented] (KAFKA-4477) Node reduces its ISR to itself, and doesn't recover. Other nodes do not take leadership, cluster remains sick until node is restarted.

[ https://issues.apache.org/jira/browse/KAFKA-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15746799#comment-15746799 ] Apurva Mehta commented on KAFKA-4477: - [~tdevoe], thanks for sharing all your extend broker logs

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

@Becket and @Rajini, Thanks for those comments. You raise some very astute points. I will address a subset of them here. One common thread across your emails has to do with the notion of a 'batch' of messages from the consumer's point of view. In particular, Rajini's points 12 and 16, and

[jira] [Commented] (KAFKA-4477) Node reduces its ISR to itself, and doesn't recover. Other nodes do not take leadership, cluster remains sick until node is restarted.

[ https://issues.apache.org/jira/browse/KAFKA-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743529#comment-15743529 ] Apurva Mehta commented on KAFKA-4477: - Thanks [~tdevoe], These seem to be the broker logs, and nothing

[jira] [Commented] (KAFKA-4477) Node reduces its ISR to itself, and doesn't recover. Other nodes do not take leadership, cluster remains sick until node is restarted.

[ https://issues.apache.org/jira/browse/KAFKA-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743283#comment-15743283 ] Apurva Mehta commented on KAFKA-4477: - Hi [~tdevoe]: Thanks for sharing those logs. I had a look

[jira] [Commented] (KAFKA-4477) Node reduces its ISR to itself, and doesn't recover. Other nodes do not take leadership, cluster remains sick until node is restarted.

[ https://issues.apache.org/jira/browse/KAFKA-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15734049#comment-15734049 ] Apurva Mehta commented on KAFKA-4477: - Also, could you share the values of the following metrics

[jira] [Commented] (KAFKA-4477) Node reduces its ISR to itself, and doesn't recover. Other nodes do not take leadership, cluster remains sick until node is restarted.

[ https://issues.apache.org/jira/browse/KAFKA-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733238#comment-15733238 ] Apurva Mehta commented on KAFKA-4477: - Hi [~tdevoe], [~jj83], [~michael.andre.pearce] : I had a look

[jira] [Assigned] (KAFKA-4477) Node reduces its ISR to itself, and doesn't recover. Other nodes do not take leadership, cluster remains sick until node is restarted.

[ https://issues.apache.org/jira/browse/KAFKA-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apurva Mehta reassigned KAFKA-4477: --- Assignee: Apurva Mehta > Node reduces its ISR to itself, and doesn't recover. Other nodes

Re: [VOTE] KIP-88: OffsetFetch Protocol Update

+1 (non-binding) On Wed, Dec 7, 2016 at 10:05 AM, Jason Gustafson wrote: > +1 Thanks for the KIP! > > On Wed, Dec 7, 2016 at 2:53 AM, Ismael Juma wrote: > > > Thanks for the KIP, Vahid. +1 (binding) > > > > On Mon, Dec 5, 2016 at 6:16 PM, Vahid S

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

Hi Ben, Now, on to your first question of how deal with consumer rebalances. The short answer is that the application needs to ensure that the the assignment of input partitions to appId is consistent across rebalances. For Kafka streams, they already ensure that the mapping of input partitions

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

Hi Ben, Those are both great questions. I will tackle the second one now, and address the first one a bit later. AppIds are prerequisite for using transactions, and must be consistent across across application sessions. They are the mechanism by which transaction recovery can occur across

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

Hi Jay, Thanks for your comments. Answers to some of your points are below: 2. There have been long debates about the necessity of the initTransactions method. Let's consider the options for doing without the initTransactions method: - If we do it on demand, we have to consider that the

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

ore work for the user, though > of > > course it would be transparent to for Kafka Streams users. > > > > One final note. I've described above how to get the strongest guarantees > > that this work is capable of providing in an auto-scaling environment. We > > al

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

Hi Ismael, That is a good suggestion. We did not plan to move the design to a wiki, but I think it is valuable to move at least the message format and RPC changes to the wiki. We shall do so once the design is close to final so that we do not have to edit multiple places as we iterate. Thanks,

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

ucer with the same app-id, it creates a pid and appends (app-id, > pid, > > > epoch) into the transaction log. > > > > > > What about if the app-id/pid pair already exists and we increment the > > > epoch? Should we append (app-id, pid, epoch++) to the transact

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

to the transaction log. > > What about if the app-id/pid pair already exists and we increment the > epoch? Should we append (app-id, pid, epoch++) to the transaction log? I > think we should, but step 2 doesn't mention this. > > On Wed, Nov 30, 2016 at 5:35 PM, Apurva Mehta <a

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

Thanks for your comments, let me deal with your second point regarding merging the __consumer-offsets and transactions topic. Needless to say, we considered doing this, but chose to keep them separate for the following reasons: 1. Your assumption that group.id and transaction.app.id can be

Re: [DISCUSS] KIP-96 - Add per partition metrics for in-sync and assigned replica count

+1 (non-binding) On Wed, Nov 30, 2016 at 1:20 PM, Xavier Léauté wrote: > FYI, Based on internal feedback I renamed AssignedReplicasCount to simply > be called ReplicasCount. > > On Tue, Nov 29, 2016 at 7:56 PM Neha Narkhede wrote: > > > This seems

Re: [VOTE] KIP-96 - Add per partition metrics for in-sync and replica count

+1 (non-binding) On Wed, Nov 30, 2016 at 2:00 PM, Jason Gustafson wrote: > +1. Thanks for the KIP! > > On Wed, Nov 30, 2016 at 1:47 PM, Gwen Shapira wrote: > > > +1 (binding) > > > > On Wed, Nov 30, 2016 at 1:34 PM, Xavier Léauté > >

Re: How to commit bug fix to kafka mesos framework

If the bug is in Kafka, here is the process for submitting a fix: http://kafka.apache.org/contributing If the bug is in the mesos-kafka framework, I think you should look up that project and find out how to commit a bugfix there. I think it should not be more complicated than submitting a PR.

Re: Resources about Apache Kafka

Not sure if this answers your question, but the entry point for handling the FetchRequest is here: https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/KafkaApis.scala#L436 That file has the entry points for the handlers of other requests as well. Thanks, Apurva On Sun,

[jira] [Resolved] (KAFKA-4215) Consumers miss messages during partition reassignment

[ https://issues.apache.org/jira/browse/KAFKA-4215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apurva Mehta resolved KAFKA-4215. - Resolution: Won't Fix This is the expected behavior. When the replication factor is 1, and when

[jira] [Commented] (KAFKA-4215) Consumers miss messages during partition reassignment

[ https://issues.apache.org/jira/browse/KAFKA-4215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15519566#comment-15519566 ] Apurva Mehta commented on KAFKA-4215: - Another observation: the consumer only misses messages when

[jira] [Commented] (KAFKA-4215) Consumers miss messages during partition reassignment

[ https://issues.apache.org/jira/browse/KAFKA-4215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15519348#comment-15519348 ] Apurva Mehta commented on KAFKA-4215: - Here is [~junrao]'s theory for what is going on: {quote

[jira] [Updated] (KAFKA-4214) kafka-reassign-partitions fails all the time when brokers are bounced during reassignment

[ https://issues.apache.org/jira/browse/KAFKA-4214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apurva Mehta updated KAFKA-4214: Description: Due to KAFKA-4204, we never realized that the existing system test for testing

[jira] [Created] (KAFKA-4215) Consumers miss messages during partition reassignment

Apurva Mehta created KAFKA-4215: --- Summary: Consumers miss messages during partition reassignment Key: KAFKA-4215 URL: https://issues.apache.org/jira/browse/KAFKA-4215 Project: Kafka Issue Type

[jira] [Created] (KAFKA-4214) kafka-reassign-partitions fails all the time when brokers are bounced during reassignment

Apurva Mehta created KAFKA-4214: --- Summary: kafka-reassign-partitions fails all the time when brokers are bounced during reassignment Key: KAFKA-4214 URL: https://issues.apache.org/jira/browse/KAFKA-4214

[jira] [Work started] (KAFKA-4213) Add system tests for replication throttling (KIP-73)

[ https://issues.apache.org/jira/browse/KAFKA-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-4213 started by Apurva Mehta. --- > Add system tests for replication throttling (KIP

[jira] [Created] (KAFKA-4213) Add system tests for replication throttling (KIP-73)

Apurva Mehta created KAFKA-4213: --- Summary: Add system tests for replication throttling (KIP-73) Key: KAFKA-4213 URL: https://issues.apache.org/jira/browse/KAFKA-4213 Project: Kafka Issue Type

[jira] [Comment Edited] (KAFKA-4204) KafkaService.verify_reassign_partitions is a no-op

[ https://issues.apache.org/jira/browse/KAFKA-4204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517207#comment-15517207 ] Apurva Mehta edited comment on KAFKA-4204 at 9/23/16 6:36 PM: -- Another issue

[jira] [Commented] (KAFKA-4204) KafkaService.verify_reassign_partitions is a no-op

[ https://issues.apache.org/jira/browse/KAFKA-4204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517207#comment-15517207 ] Apurva Mehta commented on KAFKA-4204: - Another issue with `KafkaService.verify_reassign_partitions

[jira] [Updated] (KAFKA-4204) KafkaService.verify_reassign_partitions is a no-op

[ https://issues.apache.org/jira/browse/KAFKA-4204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apurva Mehta updated KAFKA-4204: Assignee: Apurva Mehta > KafkaService.verify_reassign_partitions is a no

[jira] [Created] (KAFKA-4204) KafkaService.verify_reassign_partitions is a no-op

Apurva Mehta created KAFKA-4204: --- Summary: KafkaService.verify_reassign_partitions is a no-op Key: KAFKA-4204 URL: https://issues.apache.org/jira/browse/KAFKA-4204 Project: Kafka Issue Type

<    1   2   3   4   5