[jira] [Commented] (KAFKA-2720) Periodic purging groups in the coordinator

[ https://issues.apache.org/jira/browse/KAFKA-2720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15299416#comment-15299416 ] ASF GitHub Bot commented on KAFKA-2720: --- GitHub user hachikuji opened a pull request:

[GitHub] kafka pull request: KAFKA-2720 [WIP]: expire group metadata when a...

GitHub user hachikuji opened a pull request: https://github.com/apache/kafka/pull/1427 KAFKA-2720 [WIP]: expire group metadata when all offsets have expired You can merge this pull request into a Git repository by running: $ git pull https://github.com/hachikuji/kafka

Re: [VOTE] KIP-58 - Make Log Compaction Point Configurable

+1 (binding) Agreed that the log.cleaner.compaction.delay.ms is probably a better name, and consistent with log.segment.delete.delay.ms. Checked configs for other suffixes that seemed reasonable and despite only appearing in that one broker config, it seems the best match. -Ewen On Tue, May 24,

Kafka server contains messages but consumer cannot receive any(Producer used compression.type = snappy)

Hi dev, Kafka version: 0.9.0 language: Java When using kafka, I can set a codec by setting the* compression.type=snappy *property of my kafka producer. Suppose I use snappy compression in my producer, and i can see the message use kafkaMonitor. But when I consuming the messages from kafka

Re: [VOTE] KIP-58 - Make Log Compaction Point Configurable

I'm +1 on the concept. As with others I think the core challenge is to express this in an intuitive way, and carry the same terminology across the docs, the configs, and docstrings for the configs. Pictures would help. -Jay On Tue, May 24, 2016 at 6:54 PM, James Cheng

Jenkins build is back to normal : kafka-trunk-jdk8 #647

See

Re: [VOTE] KIP-58 - Make Log Compaction Point Configurable

I'm not sure what are the rules for who is allowed to vote, but I'm: +1 (non-binding) on the proposal I agree that the "log.cleaner.min.compaction.lag.ms" name is a little confusing. I like Becket's "log.cleaner.compaction.delay.ms", or something similar. The KIP describes it as the portion of

[GitHub] kafka pull request: Setting broker state as running after publishi...

GitHub user theduderog opened a pull request: https://github.com/apache/kafka/pull/1426 Setting broker state as running after publishing to ZK @junrao Currently, the broker state is set to running before it registers itself in ZooKeeper. This is too early in the broker

Re: [VOTE] KIP-58 - Make Log Compaction Point Configurable

+1 (non-binding) on the proposal. Just a minor suggestion. I am wondering should we change the config name to " log.cleaner.compaction.delay.ms"? The first glance at the configuration name is a little confusing. I was thinking do we have a "max" lag? And is this "lag" a bad thing? Thanks,

[jira] [Commented] (KAFKA-3370) Add options to auto.offset.reset to reset offsets upon initialization only

[ https://issues.apache.org/jira/browse/KAFKA-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15299252#comment-15299252 ] Gwen Shapira commented on KAFKA-3370: - Not necessarily efficiency - maybe correctness of some kind.

Build failed in Jenkins: kafka-trunk-jdk7 #1311

See Changes: [cshapi] Minor: Fix ps command example in docs [cshapi] KAFKA-3683; Add file descriptor recommendation to ops guide [cshapi] MINOR: Add virtual env to Kafka system test README.md

[jira] [Commented] (KAFKA-3370) Add options to auto.offset.reset to reset offsets upon initialization only

[ https://issues.apache.org/jira/browse/KAFKA-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15299249#comment-15299249 ] Vahid Hashemian commented on KAFKA-3370: [~gwenshap] Thanks for your feedback, If I understand you

[GitHub] kafka pull request: MINOR: Add virtual env to Kafka system test RE...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/1346 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Resolved] (KAFKA-3683) Add file descriptor recommendation to ops guide

[ https://issues.apache.org/jira/browse/KAFKA-3683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira resolved KAFKA-3683. - Resolution: Fixed Fix Version/s: 0.10.0.1 0.10.1.0 Issue resolved by

[jira] [Commented] (KAFKA-3683) Add file descriptor recommendation to ops guide

[ https://issues.apache.org/jira/browse/KAFKA-3683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15299230#comment-15299230 ] ASF GitHub Bot commented on KAFKA-3683: --- Github user asfgit closed the pull request at:

[GitHub] kafka pull request: KAFKA-3683: Add file descriptor recommendation...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/1353 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Build failed in Jenkins: kafka-trunk-jdk8 #646

See Changes: [cshapi] Minor: Fix ps command example in docs -- Started by an SCM change [EnvInject] - Loading node environment variables. Building remotely on ubuntu-2 (docker Ubuntu ubuntu

[GitHub] kafka pull request: Update doc

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/1386 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Commented] (KAFKA-3737) Closing connection during produce request should be log with WARN level.

[ https://issues.apache.org/jira/browse/KAFKA-3737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15299191#comment-15299191 ] Gwen Shapira commented on KAFKA-3737: - I don't mind bumping up the LOG level, but I don't think this

[jira] [Commented] (KAFKA-3736) Add http metrics reporter

[ https://issues.apache.org/jira/browse/KAFKA-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15299164#comment-15299164 ] Gwen Shapira commented on KAFKA-3736: - Any reason not to leave this as a github project (like

Jenkins build is back to normal : kafka-0.10.0-jdk7 #106

See

[jira] [Commented] (KAFKA-3370) Add options to auto.offset.reset to reset offsets upon initialization only

[ https://issues.apache.org/jira/browse/KAFKA-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15299148#comment-15299148 ] Gwen Shapira commented on KAFKA-3370: - actually, it is more complicated... suppose that you have

Re: [VOTE] KIP-58 - Make Log Compaction Point Configurable

+1 (binding) Thanks for responding to all my original concerns in the discussion thread. On Tue, May 24, 2016 at 1:37 PM, Eric Wasserman wrote: > Hi, > > I would like to begin voting on KIP-58 - Make Log Compaction Point > Configurable > > KIP-58 is here: < > >

Re: [DISCUSS] scalability limits in the coordinator

Regarding the change to the assignment field. It would be a protocol bump, otherwise consumers will not know how to parse the bytes the broker is returning, right? Or did I misunderstand the suggestion? On Tue, May 24, 2016 at 2:52 PM, Guozhang Wang wrote: > I think for just

[jira] [Commented] (KAFKA-3708) Rethink exception handling in KafkaStreams

[ https://issues.apache.org/jira/browse/KAFKA-3708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15299107#comment-15299107 ] Guozhang Wang commented on KAFKA-3708: -- Related commit to this ticket:

Re: Kafka Connect connector monitoring tasks via offset storage reader

On May 24, 2016 at 4:28:41 PM, Liquan Pei (liquan...@gmail.com) wrote: Hi Randall,  This is interesting. Essentially we can track the progress by getting data  from offset storage. What are the use cases you have in mind that uses the  offsets of source partitions? I can imagine that

[jira] [Commented] (KAFKA-3745) Consider adding join key to ValueJoiner interface

[ https://issues.apache.org/jira/browse/KAFKA-3745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15299045#comment-15299045 ] Guozhang Wang commented on KAFKA-3745: -- Let's leave this ticket open for a while in case we see more

Re: [DISCUSS] scalability limits in the coordinator

I think for just solving issue 1), Jun's suggestion is sufficient and simple. So I'd prefer that approach. In addition, Jason's optimization on the assignment field would be good for 2) and 3) as well, and I like that optimization for its simplicity and no format change as well. And in the future

[jira] [Commented] (KAFKA-3752) Provide a way for KStreams to recover from unclean shutdown

[ https://issues.apache.org/jira/browse/KAFKA-3752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15298949#comment-15298949 ] Guozhang Wang commented on KAFKA-3752: -- We are adding locks to the directory only for the case where

Re: Kafka Connect connector monitoring tasks via offset storage reader

Hi Randall, This is interesting. Essentially we can track the progress by getting data from offset storage. What are the use cases you have in mind that uses the offsets of source partitions? I can imagine that comparing the source offsets for the new data and already delivered data and make some

[jira] [Updated] (KAFKA-3752) Provide a way for KStreams to recover from unclean shutdown

[ https://issues.apache.org/jira/browse/KAFKA-3752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-3752: - Labels: architecture (was: ) > Provide a way for KStreams to recover from unclean shutdown >

[jira] [Commented] (KAFKA-3728) EndToEndAuthorizationTest offsets_topic misconfigured

[ https://issues.apache.org/jira/browse/KAFKA-3728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15298907#comment-15298907 ] ASF GitHub Bot commented on KAFKA-3728: --- GitHub user edoardocomar opened a pull request:

[GitHub] kafka pull request: KAFKA-3728 EndToEndAuthorizationTest offsets_t...

GitHub user edoardocomar opened a pull request: https://github.com/apache/kafka/pull/1425 KAFKA-3728 EndToEndAuthorizationTest offsets_topic misconfigured Set OffsetsTopicReplicationFactorProp to 3 like MinInSyncReplicasProp Earlier set acl for admin to read all

[jira] [Commented] (KAFKA-1573) Transient test failures on LogTest.testCorruptLog

[ https://issues.apache.org/jira/browse/KAFKA-1573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15298901#comment-15298901 ] Gwen Shapira commented on KAFKA-1573: - [~guozhang] is this still failing? > Transient test failures

[jira] [Commented] (KAFKA-1211) Hold the produce request with ack > 1 in purgatory until replicas' HW has larger than the produce offset

[ https://issues.apache.org/jira/browse/KAFKA-1211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15298900#comment-15298900 ] Gwen Shapira commented on KAFKA-1211: - [~junrao] is this still an issue? > Hold the produce request

[jira] [Updated] (KAFKA-1011) Decompression and re-compression on MirrorMaker could result in messages being dropped in the pipeline

[ https://issues.apache.org/jira/browse/KAFKA-1011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-1011: Resolution: Fixed Fix Version/s: (was: 0.10.1.0) Status: Resolved (was: Patch

[jira] [Updated] (KAFKA-1190) create a draw performance graph script

[ https://issues.apache.org/jira/browse/KAFKA-1190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-1190: Resolution: Won't Fix Fix Version/s: (was: 0.10.1.0) Status: Resolved (was:

[jira] [Resolved] (KAFKA-485) Support MacOS for this test framework

[ https://issues.apache.org/jira/browse/KAFKA-485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira resolved KAFKA-485. Resolution: Fixed Fix Version/s: (was: 0.10.1.0) related to old test framework which no

subscription request

[jira] [Resolved] (KAFKA-174) Add performance suite for Kafka

[ https://issues.apache.org/jira/browse/KAFKA-174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira resolved KAFKA-174. Resolution: Fixed Fix Version/s: (was: 0.10.1.0) was done ages ago... > Add performance

[jira] [Updated] (KAFKA-175) Add helper scripts to wrap the current perf tools

[ https://issues.apache.org/jira/browse/KAFKA-175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-175: --- Resolution: Fixed Fix Version/s: (was: 0.10.1.0) Status: Resolved (was: Patch

[VOTE] KIP-58 - Make Log Compaction Point Configurable

Hi, I would like to begin voting on KIP-58 - Make Log Compaction Point Configurable KIP-58 is here: < https://cwiki.apache.org/confluence/display/KAFKA/KIP-58+-+Make+Log+Compaction+Point+Configurable > The Jira ticket KAFKA-1981 Make log compaction point configurable is here:

Kafka Connect connector monitoring tasks via offset storage reader

I have a need for one of my SourceConnector implementations to configure a bunch of tasks and, when those are all “done”, request a task reconfiguration so that it can run a single task. Think: many tasks to make snapshot of database tables, then when those are completed reconfigure itself so

Build failed in Jenkins: kafka-trunk-jdk8 #645

See Changes: [cshapi] MINOR: Fix documentation table of contents and -- Started by an SCM change [EnvInject] - Loading node environment variables. Building remotely on ubuntu-2 (docker Ubuntu

[jira] [Assigned] (KAFKA-3370) Add options to auto.offset.reset to reset offsets upon initialization only

[ https://issues.apache.org/jira/browse/KAFKA-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vahid Hashemian reassigned KAFKA-3370: -- Assignee: Vahid Hashemian > Add options to auto.offset.reset to reset offsets upon

[GitHub] kafka pull request: MINOR: Fix documentation table of contents and...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/1423 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Re: [DISCUSS] scalability limits in the coordinator

Hi Jason, There are a few problems we want to solve here: 1. The group metadata is too big to be appended to the log. 2. Reduce the memory footprint on the broker 3. Reduce the bytes transferred over the wire. To solve (1), I like your idea of having separate messages per member. The proposal

Re: Kafka KIP meeting May 24 at 11:00am PST

The following are the notes from today's KIP discussion. - KIP-58 - Make Log Compaction Point Configurable: We want to start with just a time-based configuration since there is no good usage for byte-based or message-based configuration. Eric will change the KIP and start the vote. -

[jira] [Commented] (KAFKA-3728) EndToEndAuthorizationTest offsets_topic misconfigured

[ https://issues.apache.org/jira/browse/KAFKA-3728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15298724#comment-15298724 ] ASF GitHub Bot commented on KAFKA-3728: --- Github user edoardocomar closed the pull request at:

[GitHub] kafka pull request: KAFKA-3728 EndToEndAuthorizationTest offsets_t...

Github user edoardocomar closed the pull request at: https://github.com/apache/kafka/pull/1414 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[jira] [Commented] (KAFKA-3728) EndToEndAuthorizationTest offsets_topic misconfigured

[ https://issues.apache.org/jira/browse/KAFKA-3728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15298722#comment-15298722 ] ASF GitHub Bot commented on KAFKA-3728: --- Github user edoardocomar closed the pull request at:

[jira] [Commented] (KAFKA-3728) EndToEndAuthorizationTest offsets_topic misconfigured

[ https://issues.apache.org/jira/browse/KAFKA-3728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15298723#comment-15298723 ] ASF GitHub Bot commented on KAFKA-3728: --- GitHub user edoardocomar reopened a pull request:

[GitHub] kafka pull request: KAFKA-3728 EndToEndAuthorizationTest offsets_t...

Github user edoardocomar closed the pull request at: https://github.com/apache/kafka/pull/1414 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] kafka pull request: KAFKA-3728 EndToEndAuthorizationTest offsets_t...

GitHub user edoardocomar reopened a pull request: https://github.com/apache/kafka/pull/1414 KAFKA-3728 EndToEndAuthorizationTest offsets_topic misconfigured Set OffsetsTopicReplicationFactorProp to 3 like MinInSyncReplicasProp and OffsetCommitRequiredAcksProp to 1 to avoid

[jira] [Commented] (KAFKA-3370) Add options to auto.offset.reset to reset offsets upon initialization only

[ https://issues.apache.org/jira/browse/KAFKA-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15298700#comment-15298700 ] Vahid Hashemian commented on KAFKA-3370: [~guozhang] Is this

Re: [ANNOUCE] Apache Kafka 0.10.0.0 Released

Awesome! Thanks for managing the release Gwen! On Tue, May 24, 2016 at 11:24 AM, Gwen Shapira wrote: > The Apache Kafka community is pleased to announce the release for Apache > Kafka 0.10.0.0. > This is a major release with exciting new features, including first > release

[jira] [Created] (KAFKA-3752) Provide a way for KStreams to recover from unclean shutdown

Roger Hoover created KAFKA-3752: --- Summary: Provide a way for KStreams to recover from unclean shutdown Key: KAFKA-3752 URL: https://issues.apache.org/jira/browse/KAFKA-3752 Project: Kafka

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

It seems that the links to images in the KIP are broken. Liquan On Tue, May 24, 2016 at 9:33 AM, parth brahmbhatt < brahmbhatt.pa...@gmail.com> wrote: > 110. What does getDelegationTokenAs mean? > In the current proposal we only allow a user to get delegation token for > the identity that it

Re: [ANNOUCE] Apache Kafka 0.10.0.0 Released

I had an offline chat with Gwen and the 0.10.0.0-rc6 tag is the right one. I've updated the 0.10.0.0 tag to be the same: https://github.com/apache/kafka/tree/0.10.0.0 Ismael On Tue, May 24, 2016 at 6:04 PM, Ismael Juma wrote: > Hmm, sorry. The tag seems wrong. The commit

Re: [ANNOUCE] Apache Kafka 0.10.0.0 Released

Great!!! cheers :) Thanks, Mayuresh On Tue, May 24, 2016 at 10:07 AM, Ismael Juma wrote: > Awesome, thanks for running the release Gwen. :) > > Ismael > > On Tue, May 24, 2016 at 5:24 PM, Gwen Shapira wrote: > > > The Apache Kafka community is pleased

Re: [ANNOUCE] Apache Kafka 0.10.0.0 Released

Awesome, thanks for running the release Gwen. :) Ismael On Tue, May 24, 2016 at 5:24 PM, Gwen Shapira wrote: > The Apache Kafka community is pleased to announce the release for Apache > Kafka 0.10.0.0. > This is a major release with exciting new features, including first >

Re: [ANNOUCE] Apache Kafka 0.10.0.0 Released

Hmm, sorry. The tag seems wrong. The commit you linked Tom seems the correct one: https://github.com/apache/kafka/commit/b8642491e78c5a137f5012e31d347c01f3b02339 Gwen, is this right? Ismael On Tue, May 24, 2016 at 6:03 PM, Ismael Juma wrote: > Hi Tom, > > The official

Re: [ANNOUCE] Apache Kafka 0.10.0.0 Released

Hi Tom, The official commit can always be found via the relevant Git tag: https://github.com/apache/kafka/tree/0.10.0.0 https://github.com/apache/kafka/commit/1b5879653e0d956c79556301d1d11987baf6f2d7 Ismael On Tue, May 24, 2016 at 5:57 PM, Tom Crayford wrote: > Can I

[GitHub] kafka-site pull request: fix typo in site_name

Github user ijuma commented on the pull request: https://github.com/apache/kafka-site/pull/15#issuecomment-221336904 Thanks for the PR, merged. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] kafka-site pull request: fix typo in site_name

Github user asfgit closed the pull request at: https://github.com/apache/kafka-site/pull/15 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Re: [ANNOUCE] Apache Kafka 0.10.0.0 Released

Can I just confirm that https://github.com/apache/kafka/commit/b8642491e78c5a137f5012e31d347c01f3b02339 is the official commit for the release? The source download doesn't have the git repo and I can't see a sha anywhere in the downloaded source. On Tue, May 24, 2016 at 5:42 PM, Becket Qin

[jira] [Resolved] (KAFKA-3120) Consumer doesn't get messages from some partitions after reassign

[ https://issues.apache.org/jira/browse/KAFKA-3120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-3120. Resolution: Duplicate > Consumer doesn't get messages from some partitions after reassign >

[jira] [Commented] (KAFKA-3120) Consumer doesn't get messages from some partitions after reassign

[ https://issues.apache.org/jira/browse/KAFKA-3120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15298478#comment-15298478 ] Jason Gustafson commented on KAFKA-3120: I'm going to go ahead and close this as a duplicate of

Re: [ANNOUCE] Apache Kafka 0.10.0.0 Released

Awesome! On Tue, May 24, 2016 at 9:41 AM, Jay Kreps wrote: > Woohoo!!! :-) > > -Jay > > On Tue, May 24, 2016 at 9:24 AM, Gwen Shapira wrote: > > > The Apache Kafka community is pleased to announce the release for Apache > > Kafka 0.10.0.0. > > This is a

Re: [ANNOUCE] Apache Kafka 0.10.0.0 Released

Woohoo!!! :-) -Jay On Tue, May 24, 2016 at 9:24 AM, Gwen Shapira wrote: > The Apache Kafka community is pleased to announce the release for Apache > Kafka 0.10.0.0. > This is a major release with exciting new features, including first > release of KafkaStreams and many

Re: [ANNOUCE] Apache Kafka 0.10.0.0 Released

Gwen, Thanks for running the release! Jun On Tue, May 24, 2016 at 9:24 AM, Gwen Shapira wrote: > The Apache Kafka community is pleased to announce the release for Apache > Kafka 0.10.0.0. > This is a major release with exciting new features, including first > release of

[GitHub] kafka-site pull request: fix typo in site_name

GitHub user onurkaraman opened a pull request: https://github.com/apache/kafka-site/pull/15 fix typo in site_name You can merge this pull request into a Git repository by running: $ git pull https://github.com/onurkaraman/kafka-site asf-site Alternatively you can review and

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

110. What does getDelegationTokenAs mean? In the current proposal we only allow a user to get delegation token for the identity that it authenticated as using another mechanism, i.e. A user that authenticate using a keytab for principal us...@example.com will get delegation tokens for that user

[ANNOUCE] Apache Kafka 0.10.0.0 Released

The Apache Kafka community is pleased to announce the release for Apache Kafka 0.10.0.0. This is a major release with exciting new features, including first release of KafkaStreams and many other improvements. All of the changes in this release can be found:

Re: [DISCUSS] scalability limits in the coordinator

Hey Becket, I like your idea to store only the offset for the group metadata in memory. I think it would be safe to keep it in memory for a short time after the rebalance completes, but after that, it's only real purpose is to answer DescribeGroup requests, so your proposal makes a lot of sense

[jira] [Commented] (KAFKA-2394) Use RollingFileAppender by default in log4j.properties

[ https://issues.apache.org/jira/browse/KAFKA-2394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15298296#comment-15298296 ] Dustin Cote commented on KAFKA-2394: [~jinxing6...@126.com] are you able to make a pull request for

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

Thanks Rajini. It would probably require a separate KIP as it will introduce user visible changes. We could also update KIP-48 to have this information, but it seems cleaner to do it separately. We can discuss that in the KIP call today. Ismael On Tue, May 24, 2016 at 3:19 PM, Rajini Sivaram <

[jira] [Commented] (KAFKA-3511) Add common aggregation functions like Sum and Avg as build-ins in Kafka Streams DSL

[ https://issues.apache.org/jira/browse/KAFKA-3511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15298244#comment-15298244 ] ASF GitHub Bot commented on KAFKA-3511: --- GitHub user enothereska opened a pull request:

[GitHub] kafka pull request: KAFKA-3511: Initial commit for aggregators [Wi...

GitHub user enothereska opened a pull request: https://github.com/apache/kafka/pull/1424 KAFKA-3511: Initial commit for aggregators [WiP] Initial structure. Removed initialiser. Two simple aggregators. You can merge this pull request into a Git repository by running: $ git

[jira] [Work started] (KAFKA-3511) Add common aggregation functions like Sum and Avg as build-ins in Kafka Streams DSL

[ https://issues.apache.org/jira/browse/KAFKA-3511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-3511 started by Eno Thereska. --- > Add common aggregation functions like Sum and Avg as build-ins in Kafka >

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

Ismael, I have created a JIRA (https://issues.apache.org/jira/browse/KAFKA-3751) for adding SCRAM as a SASL mechanism. Would that need another KIP? If KIP-48 will use this mechanism, can this just be a JIRA that gets reviewed when the PR is ready? Thank you, Rajini On Tue, May 24, 2016 at 2:46

[jira] [Created] (KAFKA-3751) Add support for SASL mechanism SCRAM-SHA-256

Rajini Sivaram created KAFKA-3751: - Summary: Add support for SASL mechanism SCRAM-SHA-256 Key: KAFKA-3751 URL: https://issues.apache.org/jira/browse/KAFKA-3751 Project: Kafka Issue Type:

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

Thanks Rajini, SCRAM seems like a good candidate. Gwen had independently mentioned this as a SASL mechanism that might be useful for Kafka and I have been meaning to check it in more detail. Good to know that you are willing to contribute an implementation. Maybe we should file a separate JIRA

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

SCRAM (Salted Challenge Response Authentication Mechanism) is a better mechanism than Digest-MD5. Java doesn't come with a built-in SCRAM SaslServer or SaslClient, but I will be happy to add support in Kafka since it would be a useful mechanism to support anyway.

Jenkins build is back to normal : kafka-trunk-jdk7 #1309

See

Re: [DISCUSS] KIP-55: Secure quotas for authenticated users

Jun, I have updated the KIP based on your suggestion. Can you take a look? Thank you, Rajini On Tue, May 24, 2016 at 11:20 AM, Rajini Sivaram < rajinisiva...@googlemail.com> wrote: > Jun, > > Thank you for the review. I agree that a simple user principal based quota > is sufficient to

Build failed in Jenkins: kafka-trunk-jdk8 #644

See Changes: [ismael] KAFKA-3747; Close `RecordBatch.records` when append to batch fails -- Started by an SCM change Started by an SCM change Started by an SCM change [EnvInject] - Loading node

Build failed in Jenkins: kafka-0.10.0-jdk7 #105

See Changes: [ismael] KAFKA-3747; Close `RecordBatch.records` when append to batch fails -- Started by an SCM change Started by an SCM change Started by an SCM change Started by an SCM change

[GitHub] kafka pull request: MINOR: Fix documentation table of contents and...

GitHub user ijuma opened a pull request: https://github.com/apache/kafka/pull/1423 MINOR: Fix documentation table of contents and `BLOCK_ON_BUFFER_FULL_DOC` You can merge this pull request into a Git repository by running: $ git pull https://github.com/ijuma/kafka

[GitHub] kafka pull request: MINOR: Removed 1/2 of the hardcoded sleep time...

GitHub user enothereska opened a pull request: https://github.com/apache/kafka/pull/1422 MINOR: Removed 1/2 of the hardcoded sleep times You can merge this pull request into a Git repository by running: $ git pull https://github.com/enothereska/kafka

[jira] [Commented] (KAFKA-3218) Kafka-0.9.0.0 does not work as OSGi module

[ https://issues.apache.org/jira/browse/KAFKA-3218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15298026#comment-15298026 ] ASF GitHub Bot commented on KAFKA-3218: --- Github user rajinisivaram closed the pull request at:

[GitHub] kafka pull request: KAFKA-3218: Use static classloading for defaul...

Github user rajinisivaram closed the pull request at: https://github.com/apache/kafka/pull/888 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

Re: [DISCUSS] KIP-55: Secure quotas for authenticated users

Jun, Thank you for the review. I agree that a simple user principal based quota is sufficient to allocate broker resources fairly in a multi-user system. Hierarchical quotas proposed in the KIP currently enables clients of a user to be rate-limited as well. This allows a user to run multiple

Reporting security issues

Hi all, Since Kafka implements a number of security features, we need a procedure for reporting potential security vulnerabilities privately (as per http://www.apache.org/security/). We have added a simple page to the website that describes the procedure (thanks Flavio):

[jira] [Resolved] (KAFKA-3709) Create project security page

[ https://issues.apache.org/jira/browse/KAFKA-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-3709. Resolution: Fixed Fix Version/s: 0.10.0.0 > Create project security page >

[jira] [Commented] (KAFKA-3709) Create project security page

[ https://issues.apache.org/jira/browse/KAFKA-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15297978#comment-15297978 ] ASF GitHub Bot commented on KAFKA-3709: --- Github user asfgit closed the pull request at:

[GitHub] kafka-site pull request: KAFKA-3709: Create a project security pag...

Github user asfgit closed the pull request at: https://github.com/apache/kafka-site/pull/12 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Commented] (KAFKA-3744) Message format needs to identify serializer

[ https://issues.apache.org/jira/browse/KAFKA-3744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15297898#comment-15297898 ] Ismael Juma commented on KAFKA-3744: Hi [~davek22]. A change to the message format would require a

[jira] [Updated] (KAFKA-3747) Close `RecordBatch.records` when append to batch fails

[ https://issues.apache.org/jira/browse/KAFKA-3747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-3747: --- Resolution: Fixed Status: Resolved (was: Patch Available) Issue resolved by pull request

[jira] [Commented] (KAFKA-3747) Close `RecordBatch.records` when append to batch fails

[ https://issues.apache.org/jira/browse/KAFKA-3747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15297877#comment-15297877 ] ASF GitHub Bot commented on KAFKA-3747: --- Github user asfgit closed the pull request at:

  1   2   >