[jira] [Commented] (KAFKA-3289) Update Kafka protocol guide wiki for KIP-31 / KIP-32

[ https://issues.apache.org/jira/browse/KAFKA-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174257#comment-15174257 ] Jiangjie Qin commented on KAFKA-3289: - [~eapache] Thanks for the comments on the wiki page. I just

[jira] [Commented] (KAFKA-3289) Update Kafka protocol guide wiki for KIP-31 / KIP-32

[ https://issues.apache.org/jira/browse/KAFKA-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174318#comment-15174318 ] Evan Huus commented on KAFKA-3289: -- OK, I commented there because I didn't know this ticket existed and

[jira] [Commented] (KAFKA-2073) Replace TopicMetadata request/response with o.a.k.requests.metadata

[ https://issues.apache.org/jira/browse/KAFKA-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174271#comment-15174271 ] ASF GitHub Bot commented on KAFKA-2073: --- GitHub user hachikuji opened a pull request:

[GitHub] kafka pull request: KAFKA-2073: migrate to client-side topic metad...

GitHub user hachikuji opened a pull request: https://github.com/apache/kafka/pull/988 KAFKA-2073: migrate to client-side topic metadata request/response You can merge this pull request into a Git repository by running: $ git pull https://github.com/hachikuji/kafka KAFKA-2073

[jira] [Commented] (KAFKA-3289) Update Kafka protocol guide wiki for KIP-31 / KIP-32

[ https://issues.apache.org/jira/browse/KAFKA-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174363#comment-15174363 ] Kim Christensen commented on KAFKA-3289: Hi Jiangjie, I think it would be nice if it was clearly

[jira] [Updated] (KAFKA-3297) More optimally balanced partition assignment strategy (new consumer)

[ https://issues.apache.org/jira/browse/KAFKA-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Olson updated KAFKA-3297: Status: Patch Available (was: In Progress) > More optimally balanced partition assignment strategy

[jira] [Commented] (KAFKA-2435) More optimally balanced partition assignment strategy

[ https://issues.apache.org/jira/browse/KAFKA-2435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174235#comment-15174235 ] Andrew Olson commented on KAFKA-2435: - KIP link:

[jira] [Commented] (KAFKA-3297) More optimally balanced partition assignment strategy (new consumer)

[ https://issues.apache.org/jira/browse/KAFKA-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174237#comment-15174237 ] Andrew Olson commented on KAFKA-3297: - KIP link:

Re: [VOTE] KIP-33 - Add a time based log index to Kafka

Jiangjie, Currently, we roll a new log segment if the index is full. We can probably just do the same on the time index. This will bound the index size. 1. Sounds good. 2. I was wondering an edge case where the largest timestamp is in the oldest segment and the time index is empty is in all

[jira] [Commented] (KAFKA-3147) Memory records is not writable in MirrorMaker

[ https://issues.apache.org/jira/browse/KAFKA-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174374#comment-15174374 ] Kyle Kavanagh commented on KAFKA-3147: -- Any possibility this makes it into a 0.9.x build rather than

[GitHub] kafka pull request: KAFKA-3310: Fix for NPEs observed when throttl...

GitHub user auradkar opened a pull request: https://github.com/apache/kafka/pull/989 KAFKA-3310: Fix for NPEs observed when throttling clients. The fix basically ensures that the throttleTimeSensor is non-null before handing off to record the metric value. We also record the

[jira] [Commented] (KAFKA-3310) fetch requests can trigger repeated NPE when quota is enabled

[ https://issues.apache.org/jira/browse/KAFKA-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174365#comment-15174365 ] ASF GitHub Bot commented on KAFKA-3310: --- GitHub user auradkar opened a pull request:

[jira] [Commented] (KAFKA-3310) fetch requests can trigger repeated NPE when quota is enabled

[ https://issues.apache.org/jira/browse/KAFKA-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174366#comment-15174366 ] Aditya Auradkar commented on KAFKA-3310: [~junrao] - can you take a look? > fetch requests can

[GitHub] kafka pull request: MINOR: Move streams-exmaples source files unde...

GitHub user guozhangwang opened a pull request: https://github.com/apache/kafka/pull/992 MINOR: Move streams-exmaples source files under src folder Also remove some unused imports. You can merge this pull request into a Git repository by running: $ git pull

Re: [DISCUSS] KIP-35 - Retrieve protocol version

I don't see a use for the name - clients should be able to translate ApiKey to name for any API they support, and I'm not sure why would a client need to log anything about APIs it does not support. Am I missing something? On a related note, Magnus is currently on vacation, but he should be back

Re: [VOTE] KIP-33 - Add a time based log index to Kafka

Hi, Jiangjie, I was thinking perhaps just reusing index.interval.bytes is enough. Not sure if there is much value in adding an additional time.index.interval.ms. For 1, the timestamp index has entries of timestamp -> file position. So, there is actually no offset in the index, right? For 2,

[jira] [Updated] (KAFKA-3221) kafka-acls.sh must verify if a user has sufficient privileges to perform acls CRUD

[ https://issues.apache.org/jira/browse/KAFKA-3221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish K Singh updated KAFKA-3221: -- Resolution: Won't Fix Status: Resolved (was: Patch Available) > kafka-acls.sh must

Re: [DISCUSS] KIP-35 - Retrieve protocol version

On Tue, Mar 1, 2016 at 5:11 PM, Gwen Shapira wrote: > I don't see a use for the name - clients should be able to translate > ApiKey to name for any API they support, and I'm not sure why would a > client need to log anything about APIs it does not support. Am I > missing

[jira] [Commented] (KAFKA-3147) Memory records is not writable in MirrorMaker

[ https://issues.apache.org/jira/browse/KAFKA-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174458#comment-15174458 ] Jiangjie Qin commented on KAFKA-3147: - [~kdkavanagh] The next Kafka release will be 0.10.0. :) So

[GitHub] kafka pull request: HOTFIX: Use the correct serde classes

GitHub user guozhangwang opened a pull request: https://github.com/apache/kafka/pull/991 HOTFIX: Use the correct serde classes You can merge this pull request into a Git repository by running: $ git pull https://github.com/guozhangwang/kafka KSerde Alternatively you can

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Jay also mentioned: "Or, alternately, since deprecation has no functional impact and is just a message to developers, we could just leave it out of the protocol and just have it in release notes etc." I'm in favor of leaving it out of the protocol. I can't really see a use-case. Gwen On Mon,

[GitHub] kafka pull request: HOTFIX: Use the correct serde classes

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/991 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Works with me. I will update PR to remove this. Also, "api_name" have been pointed out as a concern. However, it can be handy for logging and similar purposes. Any take on that? On Tue, Mar 1, 2016 at 3:46 PM, Gwen Shapira wrote: > Jay also mentioned: > "Or, alternately,

Re: [DISCUSS] KIP-35 - Retrieve protocol version

Hello Dana, On Mon, Feb 29, 2016 at 4:11 PM, Dana Powers wrote: > This is a fantastic and much-needed KIP. All third-party clients have had > to deal with this issue. In my experience most clients are either declaring > they only support version broker version X, or are

[jira] [Commented] (KAFKA-2970) Both UpdateMetadataRequest.java and LeaderAndIsrRequest.java have an Endpoint class

[ https://issues.apache.org/jira/browse/KAFKA-2970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174705#comment-15174705 ] Guozhang Wang commented on KAFKA-2970: -- [~ijuma] Makes sense, we probably should make them in

[jira] [Commented] (KAFKA-3229) Root statestore is not registered with ProcessorStateManager, inner state store is registered instead

[ https://issues.apache.org/jira/browse/KAFKA-3229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173486#comment-15173486 ] Tom Dearman commented on KAFKA-3229: [~guozhang] not sure what you mean, there were no new parameters

[jira] [Commented] (KAFKA-3296) All consumer reads hang indefinately

[ https://issues.apache.org/jira/browse/KAFKA-3296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173539#comment-15173539 ] Simon Cooper commented on KAFKA-3296: - It's definitely a single broker, and a standard single-node

Re: [VOTE] KIP-33 - Add a time based log index to Kafka

Hi Jun, Rolling out a new segment when the time index is full sounds good. So both time index and offset index will be sharing the configuration of max index size. If we do that, do you think we still want to reuse index.interval.bytes? If we don't, the risk is that in some corner cases, we might

[GitHub] kafka pull request: KAFKA-3311: Prepare internal source topics bef...

GitHub user guozhangwang opened a pull request: https://github.com/apache/kafka/pull/990 KAFKA-3311: Prepare internal source topics before calling partition grouper You can merge this pull request into a Git repository by running: $ git pull

[jira] [Commented] (KAFKA-3311) Move co-partition checking to PartitionAssignor and auto-create internal topics

[ https://issues.apache.org/jira/browse/KAFKA-3311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174546#comment-15174546 ] ASF GitHub Bot commented on KAFKA-3311: --- GitHub user guozhangwang opened a pull request:

[jira] [Updated] (KAFKA-2073) Replace TopicMetadata request/response with o.a.k.requests.metadata

[ https://issues.apache.org/jira/browse/KAFKA-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson updated KAFKA-2073: --- Status: Patch Available (was: Open) > Replace TopicMetadata request/response with

[jira] [Commented] (KAFKA-3289) Update Kafka protocol guide wiki for KIP-31 / KIP-32

[ https://issues.apache.org/jira/browse/KAFKA-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174428#comment-15174428 ] Jiangjie Qin commented on KAFKA-3289: - [~kichristensen][~eapache] Yes, it looks the protocol does not

Re: Possible bug in Streams → KStreamImpl.java

Filed hotfix PR https://github.com/apache/kafka/pull/991. On Mon, Feb 29, 2016 at 3:34 PM, Guozhang Wang wrote: > Hi Avi, > > Thanks for pointing it out! And yes it is indeed a bug in the code. Could > you file a HOTFIX PR fixing this and also modify the existing unit test

[jira] [Created] (KAFKA-3313) Update Kafka protocol guide wiki page to reflect the change history of wire protocol and the corresponding Kafka version.

Jiangjie Qin created KAFKA-3313: --- Summary: Update Kafka protocol guide wiki page to reflect the change history of wire protocol and the corresponding Kafka version. Key: KAFKA-3313 URL:

[jira] [Created] (KAFKA-3314) Add CDDL license to LICENSE and NOTICE file

Jun Rao created KAFKA-3314: -- Summary: Add CDDL license to LICENSE and NOTICE file Key: KAFKA-3314 URL: https://issues.apache.org/jira/browse/KAFKA-3314 Project: Kafka Issue Type: Improvement

[jira] [Resolved] (KAFKA-2923) Improve 0.9.0 Upgrade Documents

[ https://issues.apache.org/jira/browse/KAFKA-2923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira resolved KAFKA-2923. - Resolution: Fixed > Improve 0.9.0 Upgrade Documents > > >

Build failed in Jenkins: kafka-trunk-jdk7 #1077

See Changes: [wangguoz] HOTFIX: Use the correct serde classes -- [...truncated 5623 lines...] org.apache.kafka.connect.storage.KafkaOffsetBackingStoreTest > testMissingTopic PASSED

Re: [DISCUSS] KIP-35 - Retrieve protocol version

On Tue, Mar 1, 2016 at 6:30 PM, Gwen Shapira wrote: > One more thing, the KIP actually had 3 parts: > 1. The version protocol > 2. New response on messages of wrong API key or wrong version > 3. Protocol documentation > There is a WIP patch for adding protocol docs,

[jira] [Updated] (KAFKA-3314) Add CDDL license to LICENSE and NOTICE file

[ https://issues.apache.org/jira/browse/KAFKA-3314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-3314: Priority: Blocker (was: Major) > Add CDDL license to LICENSE and NOTICE file >

[GitHub] kafka pull request: MINOR: Fixed error in test by moving commit va...

GitHub user gwenshap opened a pull request: https://github.com/apache/kafka/pull/994 MINOR: Fixed error in test by moving commit validation While looking at the failure here:

Re: [DISCUSS] KIP-35 - Retrieve protocol version

One more thing, the KIP actually had 3 parts: 1. The version protocol 2. New response on messages of wrong API key or wrong version 3. Protocol documentation I understand that you are offering to only implement part 1? But the KIP discussion and vote should still cover all three parts, they will

[GitHub] kafka pull request: MINOR: Move streams-exmaples source files unde...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/992 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Commented] (KAFKA-2944) NullPointerException in KafkaConfigStorage when config storage starts right before shutdown request

[ https://issues.apache.org/jira/browse/KAFKA-2944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174991#comment-15174991 ] Gwen Shapira commented on KAFKA-2944: - Created PR https://github.com/apache/kafka/pull/993 for partial

Re: [VOTE] KIP-33 - Add a time based log index to Kafka

Hi Jun, I see. If we only use index.interval.bytes, the time index entry will be inserted when (1) the largest timestamp is in this segment AND (2) at least index.interval.bytes have been appended since last time index entry insertion. In this case (1) becomes implicit instead of having an

Build failed in Jenkins: kafka-trunk-jdk8 #404

See Changes: [wangguoz] HOTFIX: Use the correct serde classes -- [...truncated 5045 lines...] org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > testNoTopic PASSED

Build failed in Jenkins: kafka-trunk-jdk8 #405

See Changes: [me] MINOR: Move streams-examples source files under src folder -- [...truncated 5713 lines...] org.apache.kafka.connect.storage.KafkaOffsetBackingStoreTest > testStartStop PASSED

Build failed in Jenkins: kafka-trunk-jdk7 #1078

See Changes: [me] MINOR: Move streams-examples source files under src folder -- [...truncated 5610 lines...] org.apache.kafka.connect.storage.KafkaStatusBackingStoreTest > readTaskState

[GitHub] kafka pull request: Replaced the NPE with a nicer error and clean ...

GitHub user gwenshap opened a pull request: https://github.com/apache/kafka/pull/993 Replaced the NPE with a nicer error and clean exit and added debug me… …ssage to assist with figuring this out. You can merge this pull request into a Git repository by running: $ git pull

Re: [DISCUSS] KIP-43: Kafka SASL enhancements

Also, with regards to the client flow, it says: "If sasl.mechanism is not GSSAPI, send a packet with the mechanism name to the server. Otherwise go to Step 3." It sounds like it would be more regular and simpler for clients if they always sent the mechanism, even if GSSAPI, right? The currently

Re: [DISCUSS] KIP-43: Kafka SASL enhancements

Jun, Thank you for the review. 1. With GSSAPI, the first context establishment packet starts with the byte 0x60 (tag for APPLICATION-0) followed by a variable-length encoded size, followed by various tags and contents. And the packet also contains a checksum. This is completely

Re: [DISCUSS] KIP-43: Kafka SASL enhancements

Hi Rajini, One question below. On Tue, Mar 1, 2016 at 2:47 AM, Rajini Sivaram wrote: >1. With GSSAPI, the first context establishment packet starts with the >byte 0x60 (tag for APPLICATION-0) followed by a variable-length encoded >size, followed by

[GitHub] kafka pull request: MINOR: use Vector instead of List

GitHub user xuwei-k opened a pull request: https://github.com/apache/kafka/pull/995 MINOR: use Vector instead of List `Vector#:+` is more efficient rather than `List#++` in this case. You can merge this pull request into a Git repository by running: $ git pull

[jira] [Commented] (KAFKA-3290) WorkerSourceTask testCommit transient failure

[ https://issues.apache.org/jira/browse/KAFKA-3290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15175150#comment-15175150 ] Ewen Cheslack-Postava commented on KAFKA-3290: -- [~hachikuji] Did you reproduce this locally?

[jira] [Created] (KAFKA-3316) Add Connect REST API to list available connector classes

Ewen Cheslack-Postava created KAFKA-3316: Summary: Add Connect REST API to list available connector classes Key: KAFKA-3316 URL: https://issues.apache.org/jira/browse/KAFKA-3316 Project: Kafka

[jira] [Created] (KAFKA-3315) Add Connect API to expose connector configuration info

Ewen Cheslack-Postava created KAFKA-3315: Summary: Add Connect API to expose connector configuration info Key: KAFKA-3315 URL: https://issues.apache.org/jira/browse/KAFKA-3315 Project: Kafka

[jira] [Commented] (KAFKA-3257) bootstrap-test-env.sh version check fails when grep has --colour option enabled.

[ https://issues.apache.org/jira/browse/KAFKA-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15175226#comment-15175226 ] ASF GitHub Bot commented on KAFKA-3257: --- Github user asfgit closed the pull request at:

[GitHub] kafka pull request: KAFKA-3257: disable bootstrap-test-env.sh --co...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/969 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Resolved] (KAFKA-3257) bootstrap-test-env.sh version check fails when grep has --colour option enabled.

[ https://issues.apache.org/jira/browse/KAFKA-3257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava resolved KAFKA-3257. -- Resolution: Fixed Issue resolved by pull request 969

[jira] [Commented] (KAFKA-3236) Honor Producer Configuration "block.on.buffer.full"

[ https://issues.apache.org/jira/browse/KAFKA-3236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174110#comment-15174110 ] Jiangjie Qin commented on KAFKA-3236: - Hi Thomas, >From the description of the ticket, I think your

[jira] [Comment Edited] (KAFKA-3236) Honor Producer Configuration "block.on.buffer.full"

[ https://issues.apache.org/jira/browse/KAFKA-3236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174110#comment-15174110 ] Jiangjie Qin edited comment on KAFKA-3236 at 3/1/16 6:01 PM: - Hi Thomas,

[jira] [Assigned] (KAFKA-3312) Add a offsets methods to ZkUtils and replace relevant usages

[ https://issues.apache.org/jira/browse/KAFKA-3312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vahid Hashemian reassigned KAFKA-3312: -- Assignee: Vahid Hashemian > Add a offsets methods to ZkUtils and replace relevant

Re: [DISCUSS] KIP-43: Kafka SASL enhancements

Hi Rajini, Thanks for clarifying. Comments inline. On Tue, Mar 1, 2016 at 2:21 PM, Rajini Sivaram wrote: > > Since we want to support arbitrary custom mechanisms, it feels better to > use mechanism names rather than Strings. IDs would require ensuring that >

Re: [DISCUSS] KIP-43: Kafka SASL enhancements

Ismael, Thank you, will make it clear in the docs that mechanism can be passed even for GSSAPI from 0.10.0.0. On Tue, Mar 1, 2016 at 2:59 PM, Ismael Juma wrote: > Hi Rajini, > > Thanks for clarifying. Comments inline. > > On Tue, Mar 1, 2016 at 2:21 PM, Rajini Sivaram < >

Re: [DISCUSS] KIP-43: Kafka SASL enhancements

Ismael, Thank you for the review. "*Would it be better to assign an id to each mechanism and pass that instead **of the String? That would be more space-efficient.*" Since we want to support arbitrary custom mechanisms, it feels better to use mechanism names rather than Strings. IDs would

[jira] [Commented] (KAFKA-3236) Honor Producer Configuration "block.on.buffer.full"

[ https://issues.apache.org/jira/browse/KAFKA-3236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173834#comment-15173834 ] Thomas Graves commented on KAFKA-3236: -- Thanks for the response. I saw that the config were

[jira] [Created] (KAFKA-3311) Move co-partition checking to PartitionAssignor and auto-create internal topics

Guozhang Wang created KAFKA-3311: Summary: Move co-partition checking to PartitionAssignor and auto-create internal topics Key: KAFKA-3311 URL: https://issues.apache.org/jira/browse/KAFKA-3311

[jira] [Commented] (KAFKA-3240) Replication issues on FreeBSD

[ https://issues.apache.org/jira/browse/KAFKA-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174078#comment-15174078 ] Johannes Huning commented on KAFKA-3240: >From my last runs it appears that this issue only arises

[jira] [Created] (KAFKA-3312) Add a offsets methods to ZkUtils and replace relevant usages

Grant Henke created KAFKA-3312: -- Summary: Add a offsets methods to ZkUtils and replace relevant usages Key: KAFKA-3312 URL: https://issues.apache.org/jira/browse/KAFKA-3312 Project: Kafka

[jira] [Commented] (KAFKA-3197) Producer can send message out of order even when in flight request is set to 1.

[ https://issues.apache.org/jira/browse/KAFKA-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174085#comment-15174085 ] Ismael Juma commented on KAFKA-3197: [~jjkoshy], since the patch doesn't introduce any new

Re: [DISCUSS] KIP-43: Kafka SASL enhancements

Rajini, Thanks for the explanation. For 1, this implies that we have to be careful with changing the 2-byte version in the future to avoid conflict. Could you document this in the KIP and also in the implementation? Jun On Tue, Mar 1, 2016 at 2:47 AM, Rajini Sivaram