Re: [DISCUSS] KIP-47 - Add timestamp-based log deletion policy

Hi Jun, 1. I thought more about Andrew's comment about LogAppendTime. The time-based index you are referring to is associated with KIP-33, correct? Currently my implementation is just checking the last message in a segment, so we're restricted to LogAppendTime. When the work for KIP-33 is

Re: [DISCUSS] KIP-47 - Add timestamp-based log deletion policy

Hi Becket, 1. I'm not entirely sure, log compaction by default only leaves the most recent key in. Are you proposing that we also could drop the unique key as well, depending on its timestamp? 2. Not sure if I'm qualified to answer that question. Would this be covered by existing broker logic

Re: [DISCUSS] KIP-47 - Add timestamp-based log deletion policy

Thanks for the KIP, Bill. I think it is a useful retention policy and does not require big changes. I have a few questions: 1. Does it have value to use this policy to log compaction as well? Although by definition log compaction means always keeping the latest key, but sometimes user might want

Re: Please add me to the contributor list

Thanks! looking forward to contributing -aarti On Fri, Feb 19, 2016 at 2:06 PM, Jun Rao wrote: > Aarti, > > Thanks for your interest. Added to you to the contributor list. > > Jun > > On Fri, Feb 19, 2016 at 2:03 PM, Aarti Gupta > wrote: > > > Hi

Re: [DISCUSS] KIP-47 - Add timestamp-based log deletion policy

Hi, Bill, I replied with the following comments earlier to the thread. Did you see that? Thanks for the proposal. A couple of comments. 1. It seems that this new policy should work for CreateTime as well. If a topic is configured with CreateTime, messages may not be added in strict order in the

[jira] [Commented] (KAFKA-3118) Fail test at: ProducerBounceTest. testBrokerFailure

[ https://issues.apache.org/jira/browse/KAFKA-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15155184#comment-15155184 ] aarti gupta commented on KAFKA-3118: Unable to reproduce on ubuntu, @edwardt Can you give some more

[jira] [Assigned] (KAFKA-3118) Fail test at: ProducerBounceTest. testBrokerFailure

[ https://issues.apache.org/jira/browse/KAFKA-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] aarti gupta reassigned KAFKA-3118: -- Assignee: aarti gupta (was: Jun Rao) > Fail test at: ProducerBounceTest. testBrokerFailure >

Re: [VOTE] Make next Kafka release 0.10.0.0 instead of 0.9.1.0

Hi Grant, Thanks for bringing up the compatibility and feature discussion. Ismael had similar concern about the short release interval between 0.9.0.0 and 0.10.0.0. >From wire protocol perspective, the old clients should still work well with the brokers in 0.10.0.0. So it is backward

[jira] [Created] (KAFKA-3254) Add Connect storage topic prefix for easier configuration

Jason Gustafson created KAFKA-3254: -- Summary: Add Connect storage topic prefix for easier configuration Key: KAFKA-3254 URL: https://issues.apache.org/jira/browse/KAFKA-3254 Project: Kafka

Re: [DISCUSS] KIP-47 - Add timestamp-based log deletion policy

Hi Bill, It would be good to add some information on the CreateTime versus LogAppendTime issue. Aside from that, it seems like you could start the VOTE thread. That sometimes triggers additional discussion and that is fine. Ismael On Fri, Feb 19, 2016 at 10:57 PM, Bill Warshaw

Re: [VOTE] Make next Kafka release 0.10.0.0 instead of 0.9.1.0

+1 (binding) On Fri, Feb 19, 2016 at 11:00 AM, Becket Qin wrote: > Hi All, > > We would like to start this voting thread on making next Kafka release > 0.10.0.0 instead of 0.9.1.0. > > The next Kafka release will have several significant important new > features/changes

Re: [DISCUSS] KIP-47 - Add timestamp-based log deletion policy

Hello all, What is the next step with this proposal? The work for KIP-32 that it was based off merged earlier today (https://github.com/apache/kafka/pull/764, thank you Becket). I have an implementation with tests, and I've confirmed that it actually works in a live system. Is there more

Re: Please add me to the contributor list

Aarti, Thanks for your interest. Added to you to the contributor list. Jun On Fri, Feb 19, 2016 at 2:03 PM, Aarti Gupta wrote: > Hi folks! > > Want to start contributing to Kafka, > found some newbie JIRAs such as > https://issues.apache.org/jira/browse/KAFKA-3118 > >

Please add me to the contributor list

Hi folks! Want to start contributing to Kafka, found some newbie JIRAs such as https://issues.apache.org/jira/browse/KAFKA-3118 Please, can I be added to the contributor list? My jira username is aartigupta Thanks! aarti

[jira] [Updated] (KAFKA-2971) KAFKA - Not obeying log4j settings, DailyRollingFileAppender not rolling files

[ https://issues.apache.org/jira/browse/KAFKA-2971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-2971: --- Fix Version/s: (was: 0.8.2.2) 0.9.1.0 > KAFKA - Not obeying log4j settings,

[jira] [Commented] (KAFKA-3253) Skip duplicate message size check if there is no re-compression during log appending.

[ https://issues.apache.org/jira/browse/KAFKA-3253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15154881#comment-15154881 ] Ismael Juma commented on KAFKA-3253: Yes, will do (I haven't filed it yet), > Skip duplicate message

Re: [VOTE] Make next Kafka release 0.10.0.0 instead of 0.9.1.0

+1 (non-binding) On Fri, Feb 19, 2016 at 1:03 PM, Jason Gustafson wrote: > +1 (non-binding) > > I agree with Grant on getting KIP-4 into this release if it's possible. > There are a lot of tools which depend on it. > > On Fri, Feb 19, 2016 at 12:52 PM, Ismael Juma

Re: [VOTE] Make next Kafka release 0.10.0.0 instead of 0.9.1.0

+1 (non-binding) I agree with Grant on getting KIP-4 into this release if it's possible. There are a lot of tools which depend on it. On Fri, Feb 19, 2016 at 12:52 PM, Ismael Juma wrote: > +1 (non-binding) > > Ismael > > On Fri, Feb 19, 2016 at 7:00 PM, Becket Qin

[jira] [Updated] (KAFKA-3253) Skip duplicate message size check if there is no re-compression during log appending.

[ https://issues.apache.org/jira/browse/KAFKA-3253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiangjie Qin updated KAFKA-3253: Assignee: Ismael Juma > Skip duplicate message size check if there is no re-compression during log

[jira] [Commented] (KAFKA-3253) Skip duplicate message size check if there is no re-compression during log appending.

[ https://issues.apache.org/jira/browse/KAFKA-3253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15154858#comment-15154858 ] Jiangjie Qin commented on KAFKA-3253: - [~ijuma] Thanks for letting me know. I think it makes sense to

Re: [VOTE] Make next Kafka release 0.10.0.0 instead of 0.9.1.0

+1 (non-binding) On Fri, Feb 19, 2016 at 12:52 PM, Ismael Juma wrote: > +1 (non-binding) > > Ismael > > On Fri, Feb 19, 2016 at 7:00 PM, Becket Qin wrote: > > > Hi All, > > > > We would like to start this voting thread on making next Kafka release > >

Re: [VOTE] Make next Kafka release 0.10.0.0 instead of 0.9.1.0

+1 (non-binding) Ismael On Fri, Feb 19, 2016 at 7:00 PM, Becket Qin wrote: > Hi All, > > We would like to start this voting thread on making next Kafka release > 0.10.0.0 instead of 0.9.1.0. > > The next Kafka release will have several significant important new >

[jira] [Commented] (KAFKA-3253) Skip duplicate message size check if there is no re-compression during log appending.

[ https://issues.apache.org/jira/browse/KAFKA-3253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15154843#comment-15154843 ] Ismael Juma commented on KAFKA-3253: [~becket_qin], are you planning to work on this? I am working on

[jira] [Assigned] (KAFKA-3247) kafka-topic tool can use an option to see just "unbalanced partitions"

[ https://issues.apache.org/jira/browse/KAFKA-3247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish K Singh reassigned KAFKA-3247: - Assignee: Ashish K Singh > kafka-topic tool can use an option to see just "unbalanced

Build failed in Jenkins: kafka-trunk-jdk7 #1049

See Changes: [junrao] trivial fix to authorization CLI table -- [...truncated 1479 lines...] kafka.log.LogTest > testTruncateTo PASSED kafka.log.LogTest > testCleanShutdownFile PASSED

[jira] [Commented] (KAFKA-3247) kafka-topic tool can use an option to see just "unbalanced partitions"

[ https://issues.apache.org/jira/browse/KAFKA-3247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15154743#comment-15154743 ] Gwen Shapira commented on KAFKA-3247: - Go for it, [~singhashish] > kafka-topic tool can use an option

[jira] [Updated] (KAFKA-3253) Skip duplicate message size check if there is no re-compression during log appending.

[ https://issues.apache.org/jira/browse/KAFKA-3253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiangjie Qin updated KAFKA-3253: Affects Version/s: 0.9.0.0 Fix Version/s: 0.10.0.0 > Skip duplicate message size check if

[jira] [Created] (KAFKA-3253) Skip duplicate message size check if there is no re-compression during log appending.

Jiangjie Qin created KAFKA-3253: --- Summary: Skip duplicate message size check if there is no re-compression during log appending. Key: KAFKA-3253 URL: https://issues.apache.org/jira/browse/KAFKA-3253

[jira] [Commented] (KAFKA-3251) Requesting committed offsets results in inconsistent results

[ https://issues.apache.org/jira/browse/KAFKA-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15154726#comment-15154726 ] Jason Gustafson commented on KAFKA-3251: [~dimitrij.denisse...@blacksquaremedia.com] Thanks for

Re: [VOTE] Make next Kafka release 0.10.0.0 instead of 0.9.1.0

+1 (non-binding) In general I am for this, but wanted to discuss a few things: *Compatibility* Given the (theoretically) short time period between this release and 0.9. I suggest we make this release as compatible as possible. Especially from the client perspective. Other open source projects

[jira] [Assigned] (KAFKA-3251) Requesting committed offsets results in inconsistent results

[ https://issues.apache.org/jira/browse/KAFKA-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson reassigned KAFKA-3251: -- Assignee: Jason Gustafson > Requesting committed offsets results in inconsistent

[jira] [Assigned] (KAFKA-3252) compression type for a topic should be used during log compaction

[ https://issues.apache.org/jira/browse/KAFKA-3252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish K Singh reassigned KAFKA-3252: - Assignee: Ashish K Singh > compression type for a topic should be used during log

[jira] [Commented] (KAFKA-3025) KIP-31 (part 1): Add timestamp field to message, configs, and Producer/ConsumerRecord

[ https://issues.apache.org/jira/browse/KAFKA-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15154695#comment-15154695 ] Jiangjie Qin commented on KAFKA-3025: - [~junrao] Thanks a lot for the review. Sure, I will submit a

Re: [VOTE] Make next Kafka release 0.10.0.0 instead of 0.9.1.0

+1 (binding) On Fri, Feb 19, 2016 at 11:00 AM, Becket Qin wrote: > Hi All, > > We would like to start this voting thread on making next Kafka release > 0.10.0.0 instead of 0.9.1.0. > > The next Kafka release will have several significant important new > features/changes

[VOTE] Make next Kafka release 0.10.0.0 instead of 0.9.1.0

Hi All, We would like to start this voting thread on making next Kafka release 0.10.0.0 instead of 0.9.1.0. The next Kafka release will have several significant important new features/changes such as Kafka Stream, Message Format Change, Client Interceptors and several new consumer API changes,

[jira] [Created] (KAFKA-3252) compression type for a topic should be used during log compaction

Jun Rao created KAFKA-3252: -- Summary: compression type for a topic should be used during log compaction Key: KAFKA-3252 URL: https://issues.apache.org/jira/browse/KAFKA-3252 Project: Kafka Issue

[jira] [Commented] (KAFKA-3025) KIP-31 (part 1): Add timestamp field to message, configs, and Producer/ConsumerRecord

[ https://issues.apache.org/jira/browse/KAFKA-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15154642#comment-15154642 ] Jun Rao commented on KAFKA-3025: [~becket_qin], thanks again for the patch. Currently, in Log.append(), we

[GitHub] kafka pull request: KAFKA-3233: expose consumer per-topic metrics

GitHub user happymap opened a pull request: https://github.com/apache/kafka/pull/939 KAFKA-3233: expose consumer per-topic metrics In version of 0.8.2.1, the old consumer will provide the metrics reporter per-topic consumer metrics under group 'ConsumerTopicMetrics'. For example:

[jira] [Commented] (KAFKA-3233) Per topic consumer metrics missing in new consumer's metrics reporter

[ https://issues.apache.org/jira/browse/KAFKA-3233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15154624#comment-15154624 ] ASF GitHub Bot commented on KAFKA-3233: --- GitHub user happymap opened a pull request:

[GitHub] kafka pull request: MINOR: Remove usage of deprecated junit.framew...

GitHub user granthenke opened a pull request: https://github.com/apache/kafka/pull/938 MINOR: Remove usage of deprecated junit.framework.Assert You can merge this pull request into a Git repository by running: $ git pull https://github.com/granthenke/kafka junit-dep

[jira] [Commented] (KAFKA-3233) Per topic consumer metrics missing in new consumer's metrics reporter

[ https://issues.apache.org/jira/browse/KAFKA-3233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15154611#comment-15154611 ] ASF GitHub Bot commented on KAFKA-3233: --- Github user happymap closed the pull request at:

[GitHub] kafka pull request: KAFKA-3233: expose per topic consumer metrics ...

Github user happymap closed the pull request at: https://github.com/apache/kafka/pull/905 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Re: [POLL] Make next Kafka Release 0.10.0.0 instead of 0.9.1.0

Hi Ismael, I was trying to see what people think here. I will create an official [VOTE] thread. Thanks, Jiangjie (Becket) Qin On Thu, Feb 18, 2016 at 6:14 AM, Tom Graves wrote: > +1 > Tom > > On Tuesday, February 9, 2016 12:08 PM, Becket Qin < >

[ANNOUCE] Apache Kafka 0.9.0.1 Released

The Apache Kafka community is pleased to announce the release for Apache Kafka 0.9.0.1. This is a bug fix release that fixes 70 issues reported in 0.9.0.0. All of the changes in this release can be found: https://www.apache.org/dyn/closer.cgi?path=/kafka/0.9.0.1/RELEASE_NOTES.html Apache Kafka

Build failed in Jenkins: kafka-trunk-jdk7 #1048

See Changes: [junrao] KAFKA-3025; Added timetamp to Message and use relative offset. -- [...truncated 1481 lines...] kafka.log.LogTest > testTruncateTo PASSED kafka.log.LogTest >

[jira] [Commented] (KAFKA-3240) Replication issues on FreeBSD

[ https://issues.apache.org/jira/browse/KAFKA-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15154532#comment-15154532 ] Johannes Huning commented on KAFKA-3240: Let me add some more information to this as I have

Jenkins build is back to normal : kafka-trunk-jdk8 #376

See

[GitHub] kafka pull request: KAFKA-3025 Added timetamp to Message and use r...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/764 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Updated] (KAFKA-3025) KIP-31 (part 1): Add timestamp field to message, configs, and Producer/ConsumerRecord

[ https://issues.apache.org/jira/browse/KAFKA-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-3025: --- Resolution: Fixed Fix Version/s: 0.9.1.0 Status: Resolved (was: Patch Available) Issue

[jira] [Commented] (KAFKA-3025) KIP-31 (part 1): Add timestamp field to message, configs, and Producer/ConsumerRecord

[ https://issues.apache.org/jira/browse/KAFKA-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15154391#comment-15154391 ] ASF GitHub Bot commented on KAFKA-3025: --- Github user asfgit closed the pull request at:

[jira] [Closed] (KAFKA-2291) Documentation Error

[ https://issues.apache.org/jira/browse/KAFKA-2291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] UTKARSH BHATNAGAR closed KAFKA-2291. > Documentation Error > --- > > Key: KAFKA-2291 >

[jira] [Commented] (KAFKA-3241) JmxReporter produces invalid JSON when a value is Infinity

[ https://issues.apache.org/jira/browse/KAFKA-3241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15154383#comment-15154383 ] Vahid Hashemian commented on KAFKA-3241: [~babakbehzad] Can you please provide more info on how to

Re: request to become contributor

Thanks Joel, much appreciated! On Thu, Feb 18, 2016 at 4:49 PM, Joel Koshy wrote: > Done - and I left a comment on that ticket as well. > > On Thu, Feb 18, 2016 at 11:32 AM, Dustin Cote wrote: > > > Hi folks, > > > > I wanted to take a crack at

[jira] [Commented] (KAFKA-3251) Requesting committed offsets results in inconsistent results

[ https://issues.apache.org/jira/browse/KAFKA-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15154153#comment-15154153 ] Dimitrij Denissenko commented on KAFKA-3251: It's unfortunately not something I can test

[jira] [Commented] (KAFKA-3251) Requesting committed offsets results in inconsistent results

[ https://issues.apache.org/jira/browse/KAFKA-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15154104#comment-15154104 ] Ismael Juma commented on KAFKA-3251: Thanks for the report. Does this happen with trunk as well? >

[jira] [Comment Edited] (KAFKA-3251) Requesting committed offsets results in inconsistent results

[ https://issues.apache.org/jira/browse/KAFKA-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15154104#comment-15154104 ] Ismael Juma edited comment on KAFKA-3251 at 2/19/16 11:20 AM: -- Thanks for the

[GitHub] kafka pull request: MINOR: Update streams RocksDb to 4.1.0

GitHub user ijuma opened a pull request: https://github.com/apache/kafka/pull/937 MINOR: Update streams RocksDb to 4.1.0 This is the latest version in Maven even though HISTORY.md includes releases all the way to 4.5.0. You can merge this pull request into a Git repository by

[jira] [Commented] (KAFKA-2832) support exclude.internal.topics in new consumer

[ https://issues.apache.org/jira/browse/KAFKA-2832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15154077#comment-15154077 ] ASF GitHub Bot commented on KAFKA-2832: --- GitHub user vahidhashemian reopened a pull request:

[GitHub] kafka pull request: KAFKA-2832: Add a consumer config option to ex...

Github user vahidhashemian closed the pull request at: https://github.com/apache/kafka/pull/932 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[jira] [Commented] (KAFKA-2832) support exclude.internal.topics in new consumer

[ https://issues.apache.org/jira/browse/KAFKA-2832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15154076#comment-15154076 ] ASF GitHub Bot commented on KAFKA-2832: --- Github user vahidhashemian closed the pull request at:

[GitHub] kafka pull request: KAFKA-2832: Add a consumer config option to ex...

GitHub user vahidhashemian reopened a pull request: https://github.com/apache/kafka/pull/932 KAFKA-2832: Add a consumer config option to exclude internal topics A new consumer config option 'exclude.internal.topics' was added to allow excluding internal topics when wildcards are

[jira] [Created] (KAFKA-3251) Requesting committed offsets results in inconsistent results

Dimitrij Denissenko created KAFKA-3251: -- Summary: Requesting committed offsets results in inconsistent results Key: KAFKA-3251 URL: https://issues.apache.org/jira/browse/KAFKA-3251 Project: