Re: [DISCUSS] KIP-185: Make exactly once in order delivery per partition the default producer setting

Hi Jason, You make some good points. First, I agree that we could have false positives which result in an OutOfOrderSequence exception for topics with low retention where messages are removed intentionally. I 100% agree that this should be tightened up. One solution is to return the log start

Re: [DISCUSS] KIP-187 - Add cumulative count metric for all Kafka rate metrics

Hi, Rajini, Thanks for the KIP. Just a minor comment. Most existing usage of Rate doesn't pass in Count and uses SampledTotal by default. So should we provide an additional constructor like the following that uses SampledTotal? new Meter(rateMetricName, totalMetricName) Thanks, Jun On Thu,

Re: [DISCUSS] KIP-188 - Add new metrics to support health checks

Hi Rajini, About the gauges, I was thinking that the attribute would be the value (i.e. commit id or version). I understand that Kafka Metrics doesn't support this (unlike Yammer Metrics), but would it make sense to add? Ismael On Thu, Aug 17, 2017 at 2:54 PM, Rajini Sivaram

Re: [DISCUSS] KIP-188 - Add new metrics to support health checks

Hi, Rajini, 1. Yammer doesn't seem to support expiration. I guess we can use the Kafka metric in this case. Thanks, Jun On Thu, Aug 17, 2017 at 4:53 PM, Roger Hoover wrote: > Rajini, > > The table is super helpful. Thank you. > > On Thu, Aug 17, 2017 at 2:16 AM,

Build failed in Jenkins: kafka-trunk-jdk8 #1918

See Changes: [me] KAFKA-5742: support ZK chroot in system tests -- [...truncated 913.73 KB...] kafka.integration.AutoOffsetResetTest >

Re: [DISCUSS] KIP-188 - Add new metrics to support health checks

Rajini, The table is super helpful. Thank you. On Thu, Aug 17, 2017 at 2:16 AM, Rajini Sivaram wrote: > Hi Roger, > > Thank you for the review. I have added a table with the scope of errors > counted for each request. > > Regards, > > Rajini > > On Thu, Aug 17, 2017

Build failed in Jenkins: kafka-trunk-jdk8 #1917

See Changes: [jason] KAFKA-5730; Consumer should invoke async commit callback before sync -- [...truncated 913.62 KB...] kafka.integration.AutoOffsetResetTest >

Jenkins build is back to normal : kafka-trunk-jdk7 #2648

See

[GitHub] kafka pull request #3690: Backport KAFKA-5742 support ZK chroot in system te...

GitHub user xvrl opened a pull request: https://github.com/apache/kafka/pull/3690 Backport KAFKA-5742 support ZK chroot in system tests backport of #3677 You can merge this pull request into a Git repository by running: $ git pull https://github.com/xvrl/kafka zk-chroot-0.11.0

[jira] [Resolved] (KAFKA-5742) Support passing ZK chroot in system tests

[ https://issues.apache.org/jira/browse/KAFKA-5742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava resolved KAFKA-5742. -- Resolution: Fixed Fix Version/s: 1.0.0 0.11.0.1 >

[GitHub] kafka pull request #3677: KAFKA-5742 support ZK chroot in system tests

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3677 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Build failed in Jenkins: kafka-trunk-jdk8 #1916

See Changes: [wangguoz] KAFKA-5733: RocksDB bulk load with lower number of levels -- [...truncated 3.72 MB...] org.apache.kafka.common.security.scram.ScramMessagesTest >

[GitHub] kafka pull request #3666: KAFKA-5730: Consumer should invoke async commit ca...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3666 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Resolved] (KAFKA-5730) Consumer should ensure consistent ordering between sync and async offset commits

[ https://issues.apache.org/jira/browse/KAFKA-5730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-5730. Resolution: Fixed Issue resolved by pull request 3666

[jira] [Created] (KAFKA-5748) Fix console producer to set timestamp and partition

Ran Ma created KAFKA-5748: - Summary: Fix console producer to set timestamp and partition Key: KAFKA-5748 URL: https://issues.apache.org/jira/browse/KAFKA-5748 Project: Kafka Issue Type: Improvement

Build failed in Jenkins: kafka-trunk-jdk7 #2647

See Changes: [wangguoz] KAFKA-5733: RocksDB bulk load with lower number of levels -- [...truncated 912.61 KB...] kafka.tools.ConsoleConsumerTest >

[GitHub] kafka pull request #3689: fix console producer to set timestamp and partitio...

GitHub user rancp opened a pull request: https://github.com/apache/kafka/pull/3689 fix console producer to set timestamp and partition You can merge this pull request into a Git repository by running: $ git pull https://github.com/rancp/kafka consoleproducer Alternatively

[GitHub] kafka pull request #3688: KAFKA-5747: Producer snapshot loading should cover...

GitHub user hachikuji opened a pull request: https://github.com/apache/kafka/pull/3688 KAFKA-5747: Producer snapshot loading should cover schema and other errors You can merge this pull request into a Git repository by running: $ git pull https://github.com/hachikuji/kafka

[GitHub] kafka pull request #3681: KAFKA-5733: RocksDB bulk load with lower number of...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3681 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Resolved] (KAFKA-5733) System tests get exception RocksDBException: db has more levels than options.num_levels

[ https://issues.apache.org/jira/browse/KAFKA-5733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-5733. -- Resolution: Fixed Issue resolved by pull request 3681

[jira] [Resolved] (KAFKA-5537) Subscribe Earliest is not working as in 0.10.2.1

[ https://issues.apache.org/jira/browse/KAFKA-5537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Andre Pearce resolved KAFKA-5537. - Resolution: Won't Fix > Subscribe Earliest is not working as in 0.10.2.1 >

[jira] [Resolved] (KAFKA-4341) KIP-87 - Add Compaction Tombstone Flag

[ https://issues.apache.org/jira/browse/KAFKA-4341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Andre Pearce resolved KAFKA-4341. - Resolution: Later > KIP-87 - Add Compaction Tombstone Flag >

[jira] [Created] (KAFKA-5747) Broker crashes on startup when trying to parse empty snapshot files

Lukasz Mierzwa created KAFKA-5747: - Summary: Broker crashes on startup when trying to parse empty snapshot files Key: KAFKA-5747 URL: https://issues.apache.org/jira/browse/KAFKA-5747 Project: Kafka

Re: [DISCUSS] KIP-185: Make exactly once in order delivery per partition the default producer setting

Some additional points for discussion: 1. The idempotent producer is still a new feature and I doubt it has gotten much use yet (in particular since it depends on a message format upgrade). Do we feel it has reached a level of stability where we are comfortable making it the default? 2. I'm

Re: KAFKA-5726

Seems you are already added to the contributor list. Feel free to assign to yourself. Guozhang On Thu, Aug 17, 2017 at 12:46 AM, Attila Kreiner wrote: > Hi All, > > I am new here, so I picked an easy issue and created a PR: > https://github.com/apache/kafka/pull/3669 >

[GitHub] kafka pull request #3687: MINOR: add upgrade section for 1.0.0

GitHub user guozhangwang opened a pull request: https://github.com/apache/kafka/pull/3687 MINOR: add upgrade section for 1.0.0 1. Add upgrade section for 1.0.0, including Streams API changes section. 2. Add metrics name changes section. You can merge this pull request into a

Re: [DISCUSS] 0.11.0.1 bug fix release

Just a quick update. The list has reduced to 6 remaining issues: https://issues.apache.org/jira/projects/KAFKA/versions/12340632 Thanks to everyone for completing and/or moving tickets to future releases. Damian On Thu, 17 Aug 2017 at 09:50 Damian Guy wrote: > Hi

[GitHub] kafka pull request #3623: HOTFIX: state transition cherry picking for 0.10.2

Github user enothereska closed the pull request at: https://github.com/apache/kafka/pull/3623 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[jira] [Resolved] (KAFKA-5545) Kafka Stream not able to successfully restart over new broker ip

[ https://issues.apache.org/jira/browse/KAFKA-5545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-5545. -- Resolution: Duplicate Duplicated as KAFKA-5152. > Kafka Stream not able to successfully

Re: KAFKA-5723: Refactor BrokerApiVersionsCommand to use the new AdminClient

As I started on working to develop the TopicCommand tool in Java (with this PR), Andrey is working on another one and he started to use some of my classes (mostly related to handling command line arguments) just with copy/paste/modify so duplicating

KAFKA-5723: Refactor BrokerApiVersionsCommand to use the new AdminClient

Good day, I am quite new here, my apologies for being kind of pushy. We had discussion here https://issues.apache.org/jira/browse/KAFKA-5723, that having separate branch for implementing the whole change https://issues.apache.org/jira/browse/KAFKA-3268 would simplify dev process. Could some of

[DISCUSS] KIP-190 Asynchronous Java Kafka Producer

I created KIP-190 https://cwiki.apache.org/confluence/display/ARIES/KIP-190+-+Make+java+kafka+ producer+asynchronous But I put it into ARIES space because I don't have access rights for the KAFKA confluence namespace. Can you please move it? 2017-08-12 2:52 GMT+03:00 Apurva Mehta

Re: [DISCUSS] KIP-188 - Add new metrics to support health checks

Hi Ismael, Thank you for the review. 1. Agree on keeping it simple with dynamic registration and no expiry. Will wait for Jun's feedback before updating KIP. 2. I have switched to two metrics for commit-id and version (not sure if it matches what you meant). I also added the client-id tag which

[jira] [Resolved] (KAFKA-2218) reassignment tool needs to parse and validate the json

[ https://issues.apache.org/jira/browse/KAFKA-2218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-2218. -- Resolution: Duplicate PR is available for KAFKA-4914. > reassignment tool needs to parse and validate

[jira] [Resolved] (KAFKA-357) Refactor zookeeper code in KafkaZookeeper into reusable components

[ https://issues.apache.org/jira/browse/KAFKA-357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-357. - Resolution: Duplicate Zookeeper related code is getting refactored in KAFKA-5027/KAFKA-5501 > Refactor

[jira] [Resolved] (KAFKA-359) Add message constructor that takes payload as a byte buffer

[ https://issues.apache.org/jira/browse/KAFKA-359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-359. - Resolution: Fixed This has been fixed in newer Kafka versions. > Add message constructor that takes

[jira] [Resolved] (KAFKA-275) max.message.size is not enforced for compressed messages

[ https://issues.apache.org/jira/browse/KAFKA-275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-275. - Resolution: Fixed This issue is fixed in latest versions. Please reopen if the issue still exists. >

[jira] [Resolved] (KAFKA-276) Enforce max.message.size on the total message size, not just on payload size

[ https://issues.apache.org/jira/browse/KAFKA-276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-276. - Resolution: Fixed This was fixed in newer Kafka versions. > Enforce max.message.size on the total message

[jira] [Resolved] (KAFKA-236) Make 'autooffset.reset' accept a delay in addition to {smallest,largest}

[ https://issues.apache.org/jira/browse/KAFKA-236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-236. - Resolution: Fixed This can be achieved by using reset consumer group tool or

[jira] [Resolved] (KAFKA-186) no clean way to getCompressionCodec from Java-the-language

[ https://issues.apache.org/jira/browse/KAFKA-186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-186. - Resolution: Fixed CompressionType Java class added in newer Kafka version. > no clean way to

[jira] [Resolved] (KAFKA-196) Topic creation fails on large values

[ https://issues.apache.org/jira/browse/KAFKA-196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-196. - Resolution: Fixed Topic MAX_NAME_LENGTH is set to 249 is newer Kafka verions. > Topic creation fails on

[GitHub] kafka pull request #3686: [WIP] KAFKA-5738: Add cumulative count for rate me...

GitHub user rajinisivaram opened a pull request: https://github.com/apache/kafka/pull/3686 [WIP] KAFKA-5738: Add cumulative count for rate metrics Implementation of https://cwiki.apache.org/confluence/display/KAFKA/KIP-187+-+Add+cumulative+count+metric+for+all+Kafka+rate+metrics

Re: [DISCUSS] KIP-187 - Add cumulative count metric for all Kafka rate metrics

Thank you, Roger. I have changed the naming of the internal classes slightly and updated the example. The new CompoundStat is called Meter and it combines Rate and Count (the aggregate), leaving Rate as a simple stat. Will submit an initial PR to help with the review. On Thu, Aug 17, 2017 at

[GitHub] kafka pull request #3685: KAFKA-5668: fetch across stores in CompositeReadOn...

GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/3685 KAFKA-5668: fetch across stores in CompositeReadOnlyWindowStore & CompositeReadOnlySessionStore Fix range queries in `CompositeReadOnlyWindowStore` and `CompositeReadOnlySessionStore` to fetch across

[GitHub] kafka pull request #3684: KIP-81: KAFKA-4133: Bound memory usage of the Cons...

GitHub user mimaison opened a pull request: https://github.com/apache/kafka/pull/3684 KIP-81: KAFKA-4133: Bound memory usage of the Consumer This is in-progress but there should be enough to start the reviews. Although the KIP doesn't mention metrics, I think we should

Re: Using Java Kafka Producer API for nearly Real Time Applications

Hi! I created KIP https://cwiki.apache.org/confluence/display/ARIES/KIP-190+-+Make+java+kafka+producer+asynchronous But I put it into ARIES space because I don't have access rights for the KAFKA confluence namespace. Can you please move it? 2017-08-12 2:52 GMT+03:00 Apurva Mehta

[GitHub] kafka pull request #3683: KAFKA-5686: update compression docs

GitHub user omkreddy opened a pull request: https://github.com/apache/kafka/pull/3683 KAFKA-5686: update compression docs You can merge this pull request into a Git repository by running: $ git pull https://github.com/omkreddy/kafka KAFKA-5686 Alternatively you can review

Re: [DISCUSS] KIP-188 - Add new metrics to support health checks

Thanks for the KIP, Rajini. I think this is helpful too. A few minor comments. 1. About the number of metrics and expiration, if we dynamically register metrics for the error codes, the number is likely to be much lower than 30*30, probably less than 100. If we were using Kafka Metrics for this,

Re: [DISCUSS] KIP-188 - Add new metrics to support health checks

Hi Jun, Thank you for the review. 1. Makes sense. I have updated the KIP. 2. Moved to a new group ZooKeeperClient 3. It is a gauge, so it will have a single attribute called Value with a constant value of 1. Regards, Rajini On Thu, Aug 17, 2017 at 3:16 AM, Jun Rao wrote:

Re: [DISCUSS] KIP-188 - Add new metrics to support health checks

Hi Roger, Thank you for the review. I have added a table with the scope of errors counted for each request. Regards, Rajini On Thu, Aug 17, 2017 at 12:05 AM, Roger Hoover wrote: > I think it would useful to make clear somewhere for each metric, the level > at which

[jira] [Created] (KAFKA-5746) Add new metrics to support health checks

Rajini Sivaram created KAFKA-5746: - Summary: Add new metrics to support health checks Key: KAFKA-5746 URL: https://issues.apache.org/jira/browse/KAFKA-5746 Project: Kafka Issue Type: New

[jira] [Resolved] (KAFKA-2206) Add AlterConfig and DescribeConfig requests to Kafka

[ https://issues.apache.org/jira/browse/KAFKA-2206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-2206. Resolution: Duplicate Thanks [~omkreddy], closing as duplicate. > Add AlterConfig and

[jira] [Resolved] (KAFKA-5242) add max_number _of_retries to exponential backoff strategy

[ https://issues.apache.org/jira/browse/KAFKA-5242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damian Guy resolved KAFKA-5242. --- Resolution: Won't Do This has been fixed by KAFKA-5152 which has already been merged to 0.11.0.1

Re: [DISCUSS] 0.11.0.1 bug fix release

Hi Srikanth, Optimistically i'm aiming for end of next week. Though it depends on how quickly the outstanding issues are closed and any other blockers that arise. Thanks, Damian On Thu, 17 Aug 2017 at 07:59 Srikanth Sampath wrote: > Thanks Damian. What's the

KAFKA-5726

Hi All, I am new here, so I picked an easy issue and created a PR: https://github.com/apache/kafka/pull/3669 https://issues.apache.org/jira/browse/KAFKA-5726 - Please add me to the contributor list so I can update the issue in Jira. - Please give me proper access rights in Confluence so

Re: [DISCUSS] 0.11.0.1 bug fix release

Thanks Damian. What's the ballpark when 0.11.0.1 will be available? -Srikanth On Wed, Aug 16, 2017 at 5:59 PM, Damian Guy wrote: > Hi, > > It seems like it must be time for 0.11.0.1 bug fix release! > > Since the 0.11.0.0 release we've fixed 30 JIRAs that > are targeted