[jira] [Commented] (KAFKA-3556) Improve group coordinator metrics

2016-08-08 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412416#comment-15412416 ] Jun Rao commented on KAFKA-3556: It would also be useful to add metrics on the consumer side to track

[jira] [Commented] (KAFKA-1211) Hold the produce request with ack > 1 in purgatory until replicas' HW has larger than the produce offset

2016-08-08 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412133#comment-15412133 ] Jun Rao commented on KAFKA-1211: [~fpj], for (a), this is because the leader needs to first wait for the

[jira] [Commented] (KAFKA-1211) Hold the produce request with ack > 1 in purgatory until replicas' HW has larger than the produce offset

2016-08-08 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412111#comment-15412111 ] Jun Rao commented on KAFKA-1211: [~fpj], yes, the idea is to always first write the new LGS before writing

[jira] [Commented] (KAFKA-3665) Default ssl.endpoint.identification.algorithm should be https

2016-08-05 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15410268#comment-15410268 ] Jun Rao commented on KAFKA-3665: [~Ryan P], thanks for the explanation. In the common case, the client

[jira] [Resolved] (KAFKA-3518) Transient failure in ReplicationTest.test_replication_with_broker_failure with ConsumerTimeoutException

2016-08-04 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-3518. Resolution: Cannot Reproduce Closing the jira since it's likely fixed by KAFKA-3670. > Transient failure

[jira] [Commented] (KAFKA-3518) Transient failure in ReplicationTest.test_replication_with_broker_failure with ConsumerTimeoutException

2016-08-04 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15408726#comment-15408726 ] Jun Rao commented on KAFKA-3518: Took a look at the log. The following are my findings. >From report.txt,

[jira] [Created] (KAFKA-4021) system tests need to enable trace level logging for controller and state-change log

2016-08-04 Thread Jun Rao (JIRA)
Jun Rao created KAFKA-4021: -- Summary: system tests need to enable trace level logging for controller and state-change log Key: KAFKA-4021 URL: https://issues.apache.org/jira/browse/KAFKA-4021 Project: Kafka

[jira] [Created] (KAFKA-4019) LogCleaner should grow read/write buffer to max message size for the topic

2016-08-04 Thread Jun Rao (JIRA)
Jun Rao created KAFKA-4019: -- Summary: LogCleaner should grow read/write buffer to max message size for the topic Key: KAFKA-4019 URL: https://issues.apache.org/jira/browse/KAFKA-4019 Project: Kafka

[jira] [Commented] (KAFKA-4009) Data corruption or EIO leads to data loss

2016-08-04 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15407990#comment-15407990 ] Jun Rao commented on KAFKA-4009: Thanks for the info. Then, N1 only discovered the issue during log

[jira] [Commented] (KAFKA-3665) Default ssl.endpoint.identification.algorithm should be https

2016-08-04 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15407964#comment-15407964 ] Jun Rao commented on KAFKA-3665: [~Ryan P], is SNI really relevant here? According to Rajini, only the

[jira] [Commented] (KAFKA-3894) Log Cleaner thread crashes and never restarts

2016-08-03 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406970#comment-15406970 ] Jun Rao commented on KAFKA-3894: [~tcrayford-heroku], I chatted with [~jkreps] on this a bit. There are a

[jira] [Commented] (KAFKA-1211) Hold the produce request with ack > 1 in purgatory until replicas' HW has larger than the produce offset

2016-08-03 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406915#comment-15406915 ] Jun Rao commented on KAFKA-1211: [~fpj], very good questions. 1. Yes, the idea is for the follower to

[jira] [Commented] (KAFKA-3875) Transient test failure: kafka.api.SslProducerSendTest.testSendNonCompressedMessageWithCreateTime

2016-08-03 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406737#comment-15406737 ] Jun Rao commented on KAFKA-3875: I took a look at this and found a few things. 1. The

[jira] [Assigned] (KAFKA-3875) Transient test failure: kafka.api.SslProducerSendTest.testSendNonCompressedMessageWithCreateTime

2016-08-03 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao reassigned KAFKA-3875: -- Assignee: Jun Rao > Transient test failure: >

[jira] [Commented] (KAFKA-3042) updateIsr should stop after failed several times due to zkVersion issue

2016-08-02 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404901#comment-15404901 ] Jun Rao commented on KAFKA-3042: [~onurkaraman], there are a couple of things. 1. Currently when a broker

[jira] [Commented] (KAFKA-1211) Hold the produce request with ack > 1 in purgatory until replicas' HW has larger than the produce offset

2016-08-02 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404404#comment-15404404 ] Jun Rao commented on KAFKA-1211: [~fpj], for #1 and #2, there are a couple scenarios that this proposal

[jira] [Commented] (KAFKA-3042) updateIsr should stop after failed several times due to zkVersion issue

2016-08-02 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404314#comment-15404314 ] Jun Rao commented on KAFKA-3042: Thinking about this a bit more. An alternative approach is to extend

[jira] [Commented] (KAFKA-2063) Bound fetch response size

2016-08-02 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404147#comment-15404147 ] Jun Rao commented on KAFKA-2063: For #3, the main reason for people to customize partition-level fetch

[jira] [Commented] (KAFKA-4009) Data corruption or EIO leads to data loss

2016-08-01 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402740#comment-15402740 ] Jun Rao commented on KAFKA-4009: [~aganesan], thanks for reporting this. In your test, did the producer

[jira] [Commented] (KAFKA-1211) Hold the produce request with ack > 1 in purgatory until replicas' HW has larger than the produce offset

2016-08-01 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402622#comment-15402622 ] Jun Rao commented on KAFKA-1211: The following is a draft proposal. [~fpj], does that look reasonable to

[jira] [Commented] (KAFKA-2063) Bound fetch response size

2016-08-01 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402263#comment-15402263 ] Jun Rao commented on KAFKA-2063: [~nepal], thanks for the proposal. For a), which server side setting are

[jira] [Updated] (KAFKA-3924) Data loss due to halting when LEO is larger than leader's LEO

2016-07-27 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-3924: --- Resolution: Fixed Fix Version/s: 0.10.0.1 Status: Resolved (was: Patch Available) Issue

[jira] [Updated] (KAFKA-3996) ByteBufferMessageSet.writeTo() should be non-blocking

2016-07-27 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-3996: --- Resolution: Fixed Status: Resolved (was: Patch Available) Issue resolved by pull request 1669

[jira] [Updated] (KAFKA-3689) Exception when attempting to decrease connection count for address with no connections

2016-07-26 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-3689: --- Assignee: (was: Jun Rao) > Exception when attempting to decrease connection count for address with no >

[jira] [Commented] (KAFKA-3996) ByteBufferMessageSet.writeTo() should be non-blocking

2016-07-26 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394755#comment-15394755 ] Jun Rao commented on KAFKA-3996: To address this issue, we will need to get rid of the while loop and

[jira] [Created] (KAFKA-3996) ByteBufferMessageSet.writeTo() should be non-blocking

2016-07-26 Thread Jun Rao (JIRA)
Jun Rao created KAFKA-3996: -- Summary: ByteBufferMessageSet.writeTo() should be non-blocking Key: KAFKA-3996 URL: https://issues.apache.org/jira/browse/KAFKA-3996 Project: Kafka Issue Type: Bug

[jira] [Commented] (KAFKA-2205) Generalize TopicConfigManager to handle multiple entity configs

2016-07-13 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15375527#comment-15375527 ] Jun Rao commented on KAFKA-2205: [~thesquelched], thanks for reporting that. Do you want to file a

[jira] [Updated] (KAFKA-2945) CreateTopic - protocol and server side implementation

2016-07-12 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-2945: --- Resolution: Fixed Status: Resolved (was: Patch Available) Issue resolved by pull request 1489

[jira] [Commented] (KAFKA-3940) Log should check the return value of dir.mkdirs()

2016-07-11 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15372146#comment-15372146 ] Jun Rao commented on KAFKA-3940: [~imandhan], we don't need to change dir to Files. We just need to change

[jira] [Commented] (KAFKA-1464) Add a throttling option to the Kafka replication tool

2016-07-11 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15372039#comment-15372039 ] Jun Rao commented on KAFKA-1464: The controller does leader balancing. So, auto.leader.rebalance.enable

[jira] [Commented] (KAFKA-1464) Add a throttling option to the Kafka replication tool

2016-07-11 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15371583#comment-15371583 ] Jun Rao commented on KAFKA-1464: Currently, our leader balancing logic happens automatically on a per

[jira] [Updated] (KAFKA-3111) java.lang.ArithmeticException: / by zero in ConsumerPerformance

2016-07-08 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-3111: --- Resolution: Fixed Fix Version/s: 0.10.1.0 Status: Resolved (was: Patch Available) Issue

[jira] [Updated] (KAFKA-3163) KIP-33 - Add a time based log index to Kafka

2016-07-08 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-3163: --- Attachment: 00113931.timeindex 00113931.log Also, I ran the following

[jira] [Commented] (KAFKA-3940) Log should check the return value of dir.mkdirs()

2016-07-08 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368315#comment-15368315 ] Jun Rao commented on KAFKA-3940: Also, instead of using File.mkdirs(), it may be better to use

[jira] [Updated] (KAFKA-3940) Log should check the return value of dir.mkdirs()

2016-07-08 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-3940: --- Labels: newbie (was: ) > Log should check the return value of dir.mkdirs() >

[jira] [Created] (KAFKA-3940) Log should check the return value of dir.mkdirs()

2016-07-08 Thread Jun Rao (JIRA)
Jun Rao created KAFKA-3940: -- Summary: Log should check the return value of dir.mkdirs() Key: KAFKA-3940 URL: https://issues.apache.org/jira/browse/KAFKA-3940 Project: Kafka Issue Type: Bug

[jira] [Created] (KAFKA-3939) add new consumer metrics in docs

2016-07-08 Thread Jun Rao (JIRA)
Jun Rao created KAFKA-3939: -- Summary: add new consumer metrics in docs Key: KAFKA-3939 URL: https://issues.apache.org/jira/browse/KAFKA-3939 Project: Kafka Issue Type: Task Components:

[jira] [Commented] (KAFKA-3919) Broker faills to start after ungraceful shutdown due to non-monotonically incrementing offsets in logs

2016-07-08 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15367781#comment-15367781 ] Jun Rao commented on KAFKA-3919: [~BigAndy], yes, the fix could be involved and we haven't nailed down a

[jira] [Commented] (KAFKA-3894) Log Cleaner thread crashes and never restarts

2016-07-07 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366339#comment-15366339 ] Jun Rao commented on KAFKA-3894: [~wushujames], this is slightly different from KAFKA-3810. In KAFKA-3810,

[jira] [Commented] (KAFKA-3919) Broker faills to start after ungraceful shutdown due to non-monotonically incrementing offsets in logs

2016-07-06 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365046#comment-15365046 ] Jun Rao commented on KAFKA-3919: [~BigAndy], yes, you are right. We actually do index based on the first

[jira] [Commented] (KAFKA-3919) Broker faills to start after ungraceful shutdown due to non-monotonically incrementing offsets in logs

2016-07-05 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15362865#comment-15362865 ] Jun Rao commented on KAFKA-3919: [~BigAndy], thanks for the investigation and the additional information.

[jira] [Commented] (KAFKA-3915) LogCleaner IO buffers do not account for potential size difference due to message format change

2016-06-30 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15357996#comment-15357996 ] Jun Rao commented on KAFKA-3915: Yes, we could choose to only keep the old message format if converting a

[jira] [Commented] (KAFKA-3919) Broker faills to start after ungraceful shutdown due to non-monotonically incrementing offsets in logs

2016-06-29 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15355393#comment-15355393 ] Jun Rao commented on KAFKA-3919: [~BigAndy], thanks for reporting and investigating this. It seems that

[jira] [Commented] (KAFKA-3915) LogCleaner IO buffers do not account for potential size difference due to message format change

2016-06-28 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15354183#comment-15354183 ] Jun Rao commented on KAFKA-3915: The writeBuffer is set to maxMessageSize. We can't really exceed that.

[jira] [Commented] (KAFKA-3252) compression type for a topic should be used during log compaction

2016-06-28 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15354083#comment-15354083 ] Jun Rao commented on KAFKA-3252: [~omkreddy], sorry for creating the duplicated jira. I took a quick look

[jira] [Commented] (KAFKA-3810) replication of internal topics should not be limited by replica.fetch.max.bytes

2016-06-28 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15354011#comment-15354011 ] Jun Rao commented on KAFKA-3810: [~onurkaraman], [~becket_qin], thanks for the patch. I have a couple of

[jira] [Commented] (KAFKA-3915) LogCleaner IO buffers do not account for potential size difference due to message format change

2016-06-28 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353396#comment-15353396 ] Jun Rao commented on KAFKA-3915: [~twbecker], thanks for reporting this. Keeping the old message format

[jira] [Commented] (KAFKA-1429) Yet another deadlock in controller shutdown

2016-06-21 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15343716#comment-15343716 ] Jun Rao commented on KAFKA-1429: [~pengwei], yes, it's a real bug. Thanks for finding and reporting this.

[jira] [Commented] (KAFKA-3868) New producer metric record-size-avg does not provide average record size as advertised

2016-06-20 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15340565#comment-15340565 ] Jun Rao commented on KAFKA-3868: One caveat is that record() requires synchronization. So, calling it once

[jira] [Commented] (KAFKA-3827) log.message.format.version should default to inter.broker.protocol.version

2016-06-20 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15340549#comment-15340549 ] Jun Rao commented on KAFKA-3827: [~ewencp], [~ijuma], the issue is that log.message.format.version should

[jira] [Updated] (KAFKA-3868) New producer metric record-size-avg does not provide average record size as advertised

2016-06-19 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-3868: --- Assignee: (was: Jun Rao) > New producer metric record-size-avg does not provide average record size as >

[jira] [Resolved] (KAFKA-3755) tightening the offset check in ReplicaFetcherThread

2016-06-16 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-3755. Resolution: Not A Problem As notes in the PR, looks like Log.append() already has that check. See

[jira] [Commented] (KAFKA-3693) Race condition between highwatermark-checkpoint thread and handleLeaderAndIsrRequest at broker start-up

2016-06-15 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15332684#comment-15332684 ] Jun Rao commented on KAFKA-3693: [~maysamyabandeh], that by itself may not be a bad idea. We will have to

[jira] [Created] (KAFKA-3827) log.message.format.version should default to inter.broker.protocol.version

2016-06-12 Thread Jun Rao (JIRA)
Jun Rao created KAFKA-3827: -- Summary: log.message.format.version should default to inter.broker.protocol.version Key: KAFKA-3827 URL: https://issues.apache.org/jira/browse/KAFKA-3827 Project: Kafka

[jira] [Commented] (KAFKA-3693) Race condition between highwatermark-checkpoint thread and handleLeaderAndIsrRequest at broker start-up

2016-06-10 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15325510#comment-15325510 ] Jun Rao commented on KAFKA-3693: [~maysamyabandeh], thanks for confirming this. The root of the problem

[jira] [Commented] (KAFKA-1120) Controller could miss a broker state change

2016-06-10 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15325496#comment-15325496 ] Jun Rao commented on KAFKA-1120: A better way is probably for the controller to store the ZK version of

[jira] [Commented] (KAFKA-3693) Race condition between highwatermark-checkpoint thread and handleLeaderAndIsrRequest at broker start-up

2016-06-09 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323675#comment-15323675 ] Jun Rao commented on KAFKA-3693: [~maysamyabandeh], the controller detects broker failure through ZK

[jira] [Commented] (KAFKA-3806) Adjust default values of log.retention.hours and offsets.retention.minutes

2016-06-09 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323521#comment-15323521 ] Jun Rao commented on KAFKA-3806: [~dwatzke], I think the original intention is that consumers in Kafka are

[jira] [Commented] (KAFKA-3806) Adjust default values of log.retention.hours and offsets.retention.minutes

2016-06-08 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15321710#comment-15321710 ] Jun Rao commented on KAFKA-3806: [~wushujames], committing offsets on every poll is likely too frequent

[jira] [Commented] (KAFKA-3806) Adjust default values of log.retention.hours and offsets.retention.minutes

2016-06-08 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15320931#comment-15320931 ] Jun Rao commented on KAFKA-3806: offsets.retention.minutes is designed to handle the case that a consumer

[jira] [Commented] (KAFKA-1211) Hold the produce request with ack > 1 in purgatory until replicas' HW has larger than the produce offset

2016-06-07 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318799#comment-15318799 ] Jun Rao commented on KAFKA-1211: This is still an issue. It can cause data loss if the leader of a

[jira] [Commented] (KAFKA-3693) Race condition between highwatermark-checkpoint thread and handleLeaderAndIsrRequest at broker start-up

2016-06-07 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318753#comment-15318753 ] Jun Rao commented on KAFKA-3693: [~maysam], when broker 16 completes the shutdown, the controller is

[jira] [Commented] (KAFKA-3693) Race condition between highwatermark-checkpoint thread and handleLeaderAndIsrRequest at broker start-up

2016-06-06 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15316652#comment-15316652 ] Jun Rao commented on KAFKA-3693: Yes, if the controller detects that a broker is down, it will remove it

[jira] [Commented] (KAFKA-3693) Race condition between highwatermark-checkpoint thread and handleLeaderAndIsrRequest at broker start-up

2016-06-02 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313441#comment-15313441 ] Jun Rao commented on KAFKA-3693: [~maysamyabandeh], yes, I agree it would be better if the broker can be

[jira] [Updated] (KAFKA-3689) ERROR Processor got uncaught exception. (kafka.network.Processor)

2016-06-02 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-3689: --- Attachment: kafka-3689-instrumentation.patch [~buvana.rama...@nokia.com], our current logging doesn't tell

[jira] [Commented] (KAFKA-3755) tightening the offset check in ReplicaFetcherThread

2016-06-02 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313028#comment-15313028 ] Jun Rao commented on KAFKA-3755: [~imandhan], we can just add a unit test that calls log.append() with

[jira] [Resolved] (KAFKA-3682) ArrayIndexOutOfBoundsException thrown by SkimpyOffsetMap.get() when full

2016-05-27 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-3682. Resolution: Fixed Fix Version/s: 0.10.1.0 Issue resolved by pull request 1352

[jira] [Commented] (KAFKA-3689) ERROR Processor got uncaught exception. (kafka.network.Processor)

2016-05-26 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303301#comment-15303301 ] Jun Rao commented on KAFKA-3689: Hmm, interesting. I assume that all the errors are for the same client

[jira] [Created] (KAFKA-3762) Log.loadSegments() should log the message in exception

2016-05-26 Thread Jun Rao (JIRA)
Jun Rao created KAFKA-3762: -- Summary: Log.loadSegments() should log the message in exception Key: KAFKA-3762 URL: https://issues.apache.org/jira/browse/KAFKA-3762 Project: Kafka Issue Type:

[jira] [Created] (KAFKA-3755) tightening the offset check in ReplicaFetcherThread

2016-05-25 Thread Jun Rao (JIRA)
Jun Rao created KAFKA-3755: -- Summary: tightening the offset check in ReplicaFetcherThread Key: KAFKA-3755 URL: https://issues.apache.org/jira/browse/KAFKA-3755 Project: Kafka Issue Type:

[jira] [Updated] (KAFKA-3721) UpdateMetadataRequest V2 should be in API version 0.10.0-IV1

2016-05-17 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-3721: --- Resolution: Fixed Status: Resolved (was: Patch Available) Issue resolved by pull request 1400

[jira] [Commented] (KAFKA-1981) Make log compaction point configurable

2016-05-15 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15284004#comment-15284004 ] Jun Rao commented on KAFKA-1981: [~ewasserman], thanks for the KIP. This seems like a useful feature.

[jira] [Resolved] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

2016-05-15 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-3565. Resolution: Fixed Fix Version/s: 0.10.0.0 Issue resolved by pull request 1372

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

2016-05-11 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15280213#comment-15280213 ] Jun Rao commented on KAFKA-3565: Since that's an existing issue, perhaps file a new jira? > Producer's

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

2016-05-10 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15279536#comment-15279536 ] Jun Rao commented on KAFKA-3565: [~guozhang], do you want to patch the producer to use the default buffer

[jira] [Commented] (KAFKA-3693) Race condition between highwatermark-checkpoint thread and handleLeaderAndIsrRequest at broker start-up

2016-05-10 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15279310#comment-15279310 ] Jun Rao commented on KAFKA-3693: [~maysamyabandeh], yes, the logic in the controller is that when a broker

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

2016-05-08 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15275959#comment-15275959 ] Jun Rao commented on KAFKA-3565: [~becket_qin], also, it seems that you patched ProducerPerformance with

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

2016-05-08 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15275957#comment-15275957 ] Jun Rao commented on KAFKA-3565: [~becket_qin], thanks for confirming this. I guess defaulting the buffer

[jira] [Commented] (KAFKA-3665) Default ssl.endpoint.identification.algorithm should be https

2016-05-08 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15275926#comment-15275926 ] Jun Rao commented on KAFKA-3665: Interesting, the difference is that in https, if a VIP is used, all

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

2016-05-08 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15275793#comment-15275793 ] Jun Rao commented on KAFKA-3565: [~becket_qin], thanks for the latest analysis. The different data size on

[jira] [Commented] (KAFKA-3587) LogCleaner fails due to incorrect offset map computation on a replica

2016-05-08 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15275741#comment-15275741 ] Jun Rao commented on KAFKA-3587: [~guozhang], I don't think (a:4) will ever be duplicated with option 1.

[jira] [Updated] (KAFKA-3670) ControlledShutdownLeaderSelector should pick the preferred replica as the new leader, if possible

2016-05-08 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-3670: --- Resolution: Fixed Status: Resolved (was: Patch Available) Issue resolved by pull request 1338

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

2016-05-06 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15274917#comment-15274917 ] Jun Rao commented on KAFKA-3565: [~becket_qin], any new findings on the consumer performance? Thanks. >

[jira] [Commented] (KAFKA-3587) LogCleaner fails due to incorrect offset map computation on a replica

2016-05-06 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15274677#comment-15274677 ] Jun Rao commented on KAFKA-3587: [~ecomar], yes, what you described matches my expectation. Now, I am not

[jira] [Created] (KAFKA-3670) ControlledShutdownLeaderSelector should pick the preferred replica as the new leader, if possible

2016-05-06 Thread Jun Rao (JIRA)
Jun Rao created KAFKA-3670: -- Summary: ControlledShutdownLeaderSelector should pick the preferred replica as the new leader, if possible Key: KAFKA-3670 URL: https://issues.apache.org/jira/browse/KAFKA-3670

[jira] [Commented] (KAFKA-3587) LogCleaner fails due to incorrect offset map computation on a replica

2016-05-06 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15274526#comment-15274526 ] Jun Rao commented on KAFKA-3587: [~alekar], note that when copying retained messages to new log segments,

[jira] [Commented] (KAFKA-3587) LogCleaner fails due to incorrect offset map computation on a replica

2016-05-06 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15274442#comment-15274442 ] Jun Rao commented on KAFKA-3587: Yes, let's first agree upon the best approach on the jira. It would be

[jira] [Commented] (KAFKA-3587) LogCleaner fails due to incorrect offset map computation on a replica

2016-05-06 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15274372#comment-15274372 ] Jun Rao commented on KAFKA-3587: [~ecomar], thanks for the explanation. I am not sure about your first

[jira] [Commented] (KAFKA-3665) Default ssl.endpoint.identification.algorithm should be https

2016-05-06 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15274195#comment-15274195 ] Jun Rao commented on KAFKA-3665: Thanks for the patch. A couple of questions. 1. Is the requirement only

[jira] [Commented] (KAFKA-3587) LogCleaner fails due to incorrect offset map computation on a replica

2016-05-05 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15273606#comment-15273606 ] Jun Rao commented on KAFKA-3587: [~alekar], I think [~ecomar] is suggesting a potentially better approach.

[jira] [Commented] (KAFKA-3587) LogCleaner fails due to incorrect offset map computation on a replica

2016-05-05 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15273457#comment-15273457 ] Jun Rao commented on KAFKA-3587: [~alekar] and [~liquanpei], thanks for both your patches. Overall, I feel

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

2016-05-05 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15272750#comment-15272750 ] Jun Rao commented on KAFKA-3565: [~becket_qin], thanks for the latest consumer results. Yes, the snappy

[jira] [Updated] (KAFKA-725) Broker Exception: Attempt to read with a maximum offset less than start offset

2016-05-05 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-725: -- Priority: Blocker (was: Major) > Broker Exception: Attempt to read with a maximum offset less than start offset

[jira] [Commented] (KAFKA-3173) Error while moving some partitions to OnlinePartition state

2016-05-05 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15272603#comment-15272603 ] Jun Rao commented on KAFKA-3173: [~fpj], yes, I agree that the lock there is confusing. Most of the time,

[jira] [Commented] (KAFKA-3330) Truncate log cleaner offset checkpoint if the log is truncated

2016-05-05 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15272621#comment-15272621 ] Jun Rao commented on KAFKA-3330: If you see the issue now, the simplest thing is to remove the affected

[jira] [Commented] (KAFKA-725) Broker Exception: Attempt to read with a maximum offset less than start offset

2016-05-05 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15272591#comment-15272591 ] Jun Rao commented on KAFKA-725: --- Yes, I agree. If the requested offset is > MaxOffset, it's better to just

[jira] [Reopened] (KAFKA-725) Broker Exception: Attempt to read with a maximum offset less than start offset

2016-05-04 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao reopened KAFKA-725: --- Reopen this jira since the fix exposes a new issue. When the leader switches (say due to leader balancing), the

[jira] [Resolved] (KAFKA-3652) Return error response for unsupported version of ApiVersionsRequest

2016-05-04 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-3652. Resolution: Fixed Committed the PR to 0.10.0. > Return error response for unsupported version of

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

2016-05-04 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15271185#comment-15271185 ] Jun Rao commented on KAFKA-3565: [~becket_qin], thanks for the results. As I was looking at the results

[jira] [Commented] (KAFKA-725) Broker Exception: Attempt to read with a maximum offset less than start offset

2016-05-04 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15270789#comment-15270789 ] Jun Rao commented on KAFKA-725: --- [~Srdo], thanks for the patch. It's still not very clear to me how a

<    1   2   3   4   5   6   7   8   9   10   >