Re: [DISCUSS] KIP-28 - Add a transform client for data processing

2015-07-27 Thread Aditya Auradkar
+1 on comparison with existing solutions. On a high level, it seems nice to have a transform library inside Kafka.. a lot of the building blocks are already there to build a stream processing framework. However the details are tricky to get right I think this discussion will get a lot more

Number of kafka topics/partitions supported per cluster of n nodes

2015-07-27 Thread Prabhjot Bharaj
Hi, I'm looking forward to a benchmark which can explain how many total number of topics and partitions can be created in a cluster of n nodes, given the message size varies between x and y bytes and how does it vary with varying heap sizes and how it affects the system performance. e.g. the

Re: [DISCUSS] KIP-28 - Add a transform client for data processing

2015-07-27 Thread Neha Narkhede
Gwen, We have a compilation of notes from comparison with other systems. They might be missing details that folks who worked on that system might be able to point out. We can share that and discuss further on the KIP call. We do hope to include a DSL since that is the most natural way of

Re: Review Request 33620: Patch for KAFKA-1690

2015-07-27 Thread Ismael Juma
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33620/#review93110 --- I did an initial pass over the code (excluding tests) and left some

Best practices - Using kafka (with http server) as source-of-truth

2015-07-27 Thread Prabhjot Bharaj
Hi Folks, I would like to understand the best practices when using kafka as the source-of-truth, given the fact that I want to pump in data to Kafka using http methods. What are the current production configurations for such a use case:- 1. Kafka-http-client - is it scalable the way Nginx is ??

Re: Review Request 33620: Patch for KAFKA-1690

2015-07-27 Thread Ismael Juma
On July 27, 2015, 1:32 p.m., Ismael Juma wrote: core/src/main/scala/kafka/api/FetchResponse.scala, line 82 https://reviews.apache.org/r/33620/diff/13/?file=1021998#file1021998line82 Casts are to be avoided in Scala, pattern matching is a better way to do this:

[jira] [Commented] (KAFKA-1682) Security for Kafka

2015-07-27 Thread Sriharsha Chintalapani (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14642865#comment-14642865 ] Sriharsha Chintalapani commented on KAFKA-1682: --- [~ijuma] Here it is

[jira] [Updated] (KAFKA-2162) Kafka Auditing functionality

2015-07-27 Thread Sriharsha Chintalapani (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sriharsha Chintalapani updated KAFKA-2162: -- Issue Type: Sub-task (was: Bug) Parent: KAFKA-1682 Kafka Auditing

[jira] [Commented] (KAFKA-1682) Security for Kafka

2015-07-27 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14642850#comment-14642850 ] Ismael Juma commented on KAFKA-1682: One of the in-scope items in the wiki page is

[jira] [Work started] (KAFKA-1686) Implement SASL/Kerberos

2015-07-27 Thread Sriharsha Chintalapani (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-1686 started by Sriharsha Chintalapani. - Implement SASL/Kerberos ---

Re: Review Request 33620: Patch for KAFKA-1690

2015-07-27 Thread Sriharsha Chintalapani
On July 27, 2015, 1:32 p.m., Ismael Juma wrote: core/src/main/scala/kafka/api/FetchResponse.scala, line 82 https://reviews.apache.org/r/33620/diff/13/?file=1021998#file1021998line82 Casts are to be avoided in Scala, pattern matching is a better way to do this:

[jira] [Commented] (KAFKA-1690) new java producer needs ssl support as a client

2015-07-27 Thread Rajasekar Elango (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14642821#comment-14642821 ] Rajasekar Elango commented on KAFKA-1690: - [~harsha_ch] Is there any documentation

[jira] [Commented] (KAFKA-1690) new java producer needs ssl support as a client

2015-07-27 Thread Sriharsha Chintalapani (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14642829#comment-14642829 ] Sriharsha Chintalapani commented on KAFKA-1690: --- [~erajasekar] I am putting

[jira] [Commented] (KAFKA-1682) Security for Kafka

2015-07-27 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14642868#comment-14642868 ] Ismael Juma commented on KAFKA-1682: [~harsha_ch], thanks for the link and for making

Re: [DISCUSS] KIP-28 - Add a transform client for data processing

2015-07-27 Thread Jay Kreps
Hey Yi, Great points. I think for some of this the most useful thing would be to get a wip prototype out that we could discuss concretely. I think Yasuhiro and Guozhang took that prototype I had done, and had some improvements. Give us a bit to get that into understandable shape so we can

[jira] [Commented] (KAFKA-2365) Copycat checklist

2015-07-27 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14643016#comment-14643016 ] Gwen Shapira commented on KAFKA-2365: - BTW. Two connectors that appeared in the KIP

[jira] [Updated] (KAFKA-2321) Introduce CONTRIBUTING.md

2015-07-27 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-2321: - Resolution: Fixed Fix Version/s: 0.8.3 Status: Resolved (was: Patch Available)

[jira] [Commented] (KAFKA-2321) Introduce CONTRIBUTING.md

2015-07-27 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14643107#comment-14643107 ] ASF GitHub Bot commented on KAFKA-2321: --- Github user asfgit closed the pull request

[jira] [Commented] (KAFKA-2349) `contributing` website page should link to Contributing Code Changes wiki page

2015-07-27 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14643129#comment-14643129 ] Guozhang Wang commented on KAFKA-2349: -- Committed to the repo, thanks!

[jira] [Updated] (KAFKA-2367) Add Copycat runtime data API

2015-07-27 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-2367: - Summary: Add Copycat runtime data API (was: Add Copycat runtime API) Add

Re: New Producer and acks configuration

2015-07-27 Thread Guozhang Wang
I think there is still a subtle difference between async with acks = 0 and async with callback, that when the #.max-inflight-requests has reached the subsequent requests cannot be sent until previous responses are returned (which could happen, for example, when the broker is slow / network issue

[jira] [Updated] (KAFKA-2368) Add Copycat standalone CLI

2015-07-27 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-2368: - Component/s: copycat Add Copycat standalone CLI --

[jira] [Updated] (KAFKA-2366) Initial patch for Copycat

2015-07-27 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-2366: - Component/s: copycat Initial patch for Copycat -

[jira] [Updated] (KAFKA-2376) Add Copycat metrics

2015-07-27 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-2376: - Component/s: copycat Add Copycat metrics ---

[jira] [Updated] (KAFKA-2373) Copycat distributed offset storage

2015-07-27 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-2373: - Component/s: copycat Copycat distributed offset storage

Re: New Producer and acks configuration

2015-07-27 Thread Mayuresh Gharat
So basically this means that with acks = 0, their is no guarantee that the message has been received by Kafka broker. I am just wondering, why would anyone be using acks = 0, since anyone using kafka and doing producer.send() would want that, their message got to kafka brokers. Also as Jay said,

Re: New Producer and acks configuration

2015-07-27 Thread Gwen Shapira
Yeah, using acks=0 should result in higher throughput since we are not limited by the roundtrip time to the broker. Btw. regarding in-flight requests: With acks = 1 (or -1), can we send a message batch to a partition before the brokers acked a previous request? Doesn't it risk getting messages

[jira] [Updated] (KAFKA-2350) Add KafkaConsumer pause capability

2015-07-27 Thread Jason Gustafson (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson updated KAFKA-2350: --- Description: There are some use cases in stream processing where it is helpful to be able to

[jira] [Commented] (KAFKA-2365) Copycat checklist

2015-07-27 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14643087#comment-14643087 ] Neha Narkhede commented on KAFKA-2365: -- Worth discussing a process for including a

[jira] [Updated] (KAFKA-2349) `contributing` website page should link to Contributing Code Changes wiki page

2015-07-27 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-2349: - Resolution: Fixed Fix Version/s: 0.8.3 Status: Resolved (was: Patch Available)

[jira] [Commented] (KAFKA-2365) Copycat checklist

2015-07-27 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14643014#comment-14643014 ] Gwen Shapira commented on KAFKA-2365: - I added a component, added you as component

[jira] [Updated] (KAFKA-2371) Add distributed coordinator implementation for Copycat

2015-07-27 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-2371: - Component/s: copycat Add distributed coordinator implementation for Copycat

[jira] [Updated] (KAFKA-2375) Implement elasticsearch Copycat sink connector

2015-07-27 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-2375: - Component/s: copycat Implement elasticsearch Copycat sink connector

[jira] [Updated] (KAFKA-2369) Add Copycat REST API

2015-07-27 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-2369: - Component/s: copycat Add Copycat REST API

[jira] [Updated] (KAFKA-2374) Implement Copycat log/file connector

2015-07-27 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-2374: - Component/s: copycat Implement Copycat log/file connector

[jira] [Updated] (KAFKA-2370) Add pause/unpause connector support

2015-07-27 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-2370: - Component/s: copycat Add pause/unpause connector support

[jira] [Updated] (KAFKA-2379) Add Copycat documentation

2015-07-27 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-2379: - Component/s: copycat Add Copycat documentation -

[jira] [Updated] (KAFKA-2378) Add Copycat embedded API

2015-07-27 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-2378: - Component/s: copycat Add Copycat embedded API

[jira] [Updated] (KAFKA-2377) Add copycat system tests

2015-07-27 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-2377: - Component/s: copycat Add copycat system tests

[jira] [Updated] (KAFKA-2372) Copycat distributed config storage

2015-07-27 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-2372: - Component/s: copycat Copycat distributed config storage

[GitHub] kafka pull request: KAFKA-2321; Introduce CONTRIBUTING.md

2015-07-27 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/97 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Updated] (KAFKA-2365) Copycat checklist

2015-07-27 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-2365: - Component/s: copycat Copycat checklist - Key:

[jira] [Updated] (KAFKA-2367) Add Copycat runtime data API

2015-07-27 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-2367: - Component/s: copycat Add Copycat runtime data API

[jira] [Commented] (KAFKA-2260) Allow specifying expected offset on produce

2015-07-27 Thread Mayuresh Gharat (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14643065#comment-14643065 ] Mayuresh Gharat commented on KAFKA-2260: I think, when 2 producers are trying to

Re: Best practices - Using kafka (with http server) as source-of-truth

2015-07-27 Thread Ewen Cheslack-Postava
Hi Prabhjot, Confluent has a REST proxy with docs that may give some guidance: http://docs.confluent.io/1.0/kafka-rest/docs/intro.html The new producer that it uses is very efficient, so you should be able to get pretty good throughput. You take a bit of a hit due to the overhead of sending data

Re: New Producer and acks configuration

2015-07-27 Thread Ewen Cheslack-Postava
If only we had some sort of system test framework with a producer performance test that we could parameterize with the different acks settings to validate these performance differences... wrt out of order: yes, with 1 in flight requests with retries, messages can get out of order. Becket had a

[jira] [Commented] (KAFKA-2303) Fix for KAFKA-2235 LogCleaner offset map overflow causes another compaction failures

2015-07-27 Thread Alexander Demidko (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14643136#comment-14643136 ] Alexander Demidko commented on KAFKA-2303: -- I think in our case we had too many

Re: [DISCUSS] Partitioning in Kafka

2015-07-27 Thread Gwen Shapira
If you are used to map-reduce patterns, this sounds like a perfectly natural way to process streams of data. Call the first consumer map-combine-log, the topic shuffle-log and the second consumer reduce-log :) I like that a lot. It works well for either embarrassingly parallel cases, or so much

Jenkins build is back to normal : KafkaPreCommit #164

2015-07-27 Thread Apache Jenkins Server
See https://builds.apache.org/job/KafkaPreCommit/164/changes

[jira] [Commented] (KAFKA-2350) Add KafkaConsumer pause capability

2015-07-27 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14643249#comment-14643249 ] Guozhang Wang commented on KAFKA-2350: -- [~becket_qin], I was not considering the

Re: Review Request 33620: Patch for KAFKA-1690

2015-07-27 Thread Dong Lin
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33620/#review93177 ---

Re: Review Request 36652: Patch for KAFKA-2351

2015-07-27 Thread Jiangjie Qin
On July 24, 2015, 4:13 p.m., Jun Rao wrote: core/src/main/scala/kafka/network/SocketServer.scala, line 264 https://reviews.apache.org/r/36652/diff/3/?file=1020607#file1020607line264 Not sure if it's better to keep the thread alive on any throwable. For unexpected exceptions, it

[jira] [Commented] (KAFKA-2364) Improve documentation for contributing to docs

2015-07-27 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14643179#comment-14643179 ] Ismael Juma commented on KAFKA-2364: Coincidentally a CONTRIBUTING.md was added today

Re: [DISCUSS] Partitioning in Kafka

2015-07-27 Thread Jason Gustafson
For a little background, the difference between this partitioner and the default one is that it breaks the deterministic mapping from key to partition. Instead, messages for a given key can end up in either of two partitions. This means that the consumer generally won't see all messages for a

Re: Kafka Consumer thoughts

2015-07-27 Thread Kartik Paramasivam
adding the open source alias. This email started off as a broader discussion around the new consumer. I was zooming into only the aspect of poll() being the only mechanism for driving the heartbeats. Yes the lag is the effect of the problem (not the problem). Monitoring the lag is important as

[jira] [Commented] (KAFKA-2350) Add KafkaConsumer pause capability

2015-07-27 Thread Jason Gustafson (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14643442#comment-14643442 ] Jason Gustafson commented on KAFKA-2350: [~becket_qin] I think that we're on the

Re: Kafka Consumer thoughts

2015-07-27 Thread Jason Gustafson
I think if we recommend a longer session timeout, then we should expose the heartbeat frequency in configuration since this generally controls how long normal rebalances will take. I think it's currently hard-coded to 3 heartbeats per session timeout. It could also be nice to have an explicit

Review Request 36858: Patch for KAFKA-2120

2015-07-27 Thread Mayuresh Gharat
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36858/ --- Review request for kafka. Bugs: KAFKA-2120

[jira] [Commented] (KAFKA-2120) Add a request timeout to NetworkClient

2015-07-27 Thread Mayuresh Gharat (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14643399#comment-14643399 ] Mayuresh Gharat commented on KAFKA-2120: Created reviewboard

[jira] [Updated] (KAFKA-2120) Add a request timeout to NetworkClient

2015-07-27 Thread Mayuresh Gharat (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayuresh Gharat updated KAFKA-2120: --- Status: Patch Available (was: Open) Add a request timeout to NetworkClient

[jira] [Updated] (KAFKA-2120) Add a request timeout to NetworkClient

2015-07-27 Thread Mayuresh Gharat (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayuresh Gharat updated KAFKA-2120: --- Attachment: KAFKA-2120.patch Add a request timeout to NetworkClient

Re: Kafka Consumer thoughts

2015-07-27 Thread Jay Kreps
Hey Kartik, Totally agree we don't want people tuning timeouts in the common case. However there are two ways to avoid this: 1. Default the timeout high 2. Put the heartbeat in a separate thread When we were doing the consumer design we discussed this tradeoff and I think the conclusion we came

Re: [DISCUSS] Partitioning in Kafka

2015-07-27 Thread Ewen Cheslack-Postava
Gwen - this is really like two steps of map reduce though, right? The first step does the partial shuffle to two partitions per key, second step does partial reduce + final full shuffle, final step does the final reduce. This strikes me as similar to partition assignment strategies in the

[jira] [Commented] (KAFKA-2350) Add KafkaConsumer pause capability

2015-07-27 Thread Jason Gustafson (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14643349#comment-14643349 ] Jason Gustafson commented on KAFKA-2350: There's one interesting implementation

Re: Review Request 36858: Patch for KAFKA-2120

2015-07-27 Thread Mayuresh Gharat
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36858/ --- (Updated July 27, 2015, 9:09 p.m.) Review request for kafka. Bugs:

[jira] [Commented] (KAFKA-2350) Add KafkaConsumer pause capability

2015-07-27 Thread Jiangjie Qin (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14643418#comment-14643418 ] Jiangjie Qin commented on KAFKA-2350: - [~hachikuji], I am with [~guozhang] that it is

Re: Kafka Consumer thoughts

2015-07-27 Thread Ewen Cheslack-Postava
Kartik, on your second point about timeouts with poll() and heartbeats, the consumer now handles this properly. KAFKA-2123 introduced a DelayedTaskQueue and that is used internally to handle processing events at the right time even if poll() is called with a large timeout. The same mechanism is

[jira] [Comment Edited] (KAFKA-2350) Add KafkaConsumer pause capability

2015-07-27 Thread Jiangjie Qin (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14643418#comment-14643418 ] Jiangjie Qin edited comment on KAFKA-2350 at 7/27/15 9:45 PM: --

Re: error while high level consumer

2015-07-27 Thread Jiangjie Qin
This is due to the zookeeper path storing the previous owner info hasn't been deleted at the moment. If the rebalance completes after retry, it should be fine. Jiangjie (Becket) Qin On Fri, Jul 24, 2015 at 6:54 PM, Kris K squareksc...@gmail.com wrote: Hi, I started seeing these errors in the

[jira] [Commented] (KAFKA-2268) New producer logs WARN if serializer supplied directly to constructor

2015-07-27 Thread Xuan Gong (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14643573#comment-14643573 ] Xuan Gong commented on KAFKA-2268: -- Looks like this is duplicate with

[jira] [Commented] (KAFKA-2130) Resource leakage in AppInfo.scala during initialization

2015-07-27 Thread Xuan Gong (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14643625#comment-14643625 ] Xuan Gong commented on KAFKA-2130: -- move {code} stream.close(); {code} to the finally

[jira] [Commented] (KAFKA-1690) new java producer needs ssl support as a client

2015-07-27 Thread Sourabh Chandak (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14643532#comment-14643532 ] Sourabh Chandak commented on KAFKA-1690: [~sriharsha] Will this patch unblock the

Re: Review Request 36858: Patch for KAFKA-2120

2015-07-27 Thread Mayuresh Gharat
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36858/ --- (Updated July 27, 2015, 10:31 p.m.) Review request for kafka. Bugs:

Re: Review Request 36858: Patch for KAFKA-2120

2015-07-27 Thread Jason Gustafson
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36858/#review93189 --- Looks pretty good overall. Found mostly trivial stuff.

[jira] [Commented] (KAFKA-1690) new java producer needs ssl support as a client

2015-07-27 Thread Sriharsha Chintalapani (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14643597#comment-14643597 ] Sriharsha Chintalapani commented on KAFKA-1690: --- [~sourabh0612] Yes. It

Re: [DISCUSS] KIP-27 - Conditional Publish

2015-07-27 Thread Jiangjie Qin
@Ewen, good point about batching. Yes, it would be tricky if we want to do a per-key conditional produce. My understanding is that the prerequisite of this KIP is: 1. Single producer for each partition. 2. Acks=-1, max.in.flight.request.per.connection=1, retries=SOME_BIG_NUMBER The major problem

Re: [DISCUSS] Partitioning in Kafka

2015-07-27 Thread Gwen Shapira
I guess it depends on whether the original producer did any map tasks or simply wrote raw data. We usually advocate writing raw data, and since we need to write it anyway, the partitioner doesn't introduce any extra hops. Its definitely useful to look at use-cases and I need to think a bit more

[jira] [Commented] (KAFKA-2120) Add a request timeout to NetworkClient

2015-07-27 Thread Mayuresh Gharat (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14643533#comment-14643533 ] Mayuresh Gharat commented on KAFKA-2120: Updated reviewboard

[jira] [Updated] (KAFKA-2120) Add a request timeout to NetworkClient

2015-07-27 Thread Mayuresh Gharat (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayuresh Gharat updated KAFKA-2120: --- Attachment: KAFKA-2120_2015-07-27_15:31:19.patch Add a request timeout to NetworkClient

Re: error while high level consumer

2015-07-27 Thread Mayuresh Gharat
Try bouncing the host that appears in the stored data section. Thanks, Mayuresh On Mon, Jul 27, 2015 at 3:41 PM, Jiangjie Qin j...@linkedin.com.invalid wrote: This is due to the zookeeper path storing the previous owner info hasn't been deleted at the moment. If the rebalance completes after

[jira] [Created] (KAFKA-2381) Possible ConcurrentModificationException while unsubscribing from a topic in new consumer

2015-07-27 Thread Ashish K Singh (JIRA)
Ashish K Singh created KAFKA-2381: - Summary: Possible ConcurrentModificationException while unsubscribing from a topic in new consumer Key: KAFKA-2381 URL: https://issues.apache.org/jira/browse/KAFKA-2381

[jira] [Commented] (KAFKA-2381) Possible ConcurrentModificationException while unsubscribing from a topic in new consumer

2015-07-27 Thread Ashish K Singh (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14643703#comment-14643703 ] Ashish K Singh commented on KAFKA-2381: --- [~gwenshap] could you take a look when you

[jira] [Updated] (KAFKA-2381) Possible ConcurrentModificationException while unsubscribing from a topic in new consumer

2015-07-27 Thread Ashish K Singh (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish K Singh updated KAFKA-2381: -- Attachment: KAFKA-2381_2015-07-27_17:56:00.patch Possible ConcurrentModificationException

[jira] [Commented] (KAFKA-2381) Possible ConcurrentModificationException while unsubscribing from a topic in new consumer

2015-07-27 Thread Ashish K Singh (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14643699#comment-14643699 ] Ashish K Singh commented on KAFKA-2381: --- Updated reviewboard

Re: Review Request 36871: Patch for KAFKA-2381

2015-07-27 Thread Aditya Auradkar
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36871/#review93215 --- core/src/test/scala/integration/kafka/api/ConsumerTest.scala (line

[jira] [Updated] (KAFKA-2381) Possible ConcurrentModificationException while unsubscribing from a topic in new consumer

2015-07-27 Thread Ashish K Singh (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish K Singh updated KAFKA-2381: -- Status: Patch Available (was: Open) Possible ConcurrentModificationException while

[jira] [Updated] (KAFKA-2381) Possible ConcurrentModificationException while unsubscribing from a topic in new consumer

2015-07-27 Thread Ashish K Singh (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish K Singh updated KAFKA-2381: -- Attachment: KAFKA-2381.patch Possible ConcurrentModificationException while unsubscribing from

Review Request 36871: Patch for KAFKA-2381

2015-07-27 Thread Ashish Singh
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36871/ --- Review request for kafka. Bugs: KAFKA-2381

[jira] [Commented] (KAFKA-313) Add JSON/CSV output and looping options to ConsumerGroupCommand

2015-07-27 Thread Ashish K Singh (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14643716#comment-14643716 ] Ashish K Singh commented on KAFKA-313: -- [~gwenshap] need help with getting this KIP to

[jira] [Updated] (KAFKA-2360) The kafka-consumer-perf-test.sh script help information print useless parameters.

2015-07-27 Thread Bo Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bo Wang updated KAFKA-2360: --- Description: Run kafka-consumer-perf-test.sh --help to show help information, but found 3 parameters

[jira] [Commented] (KAFKA-1690) new java producer needs ssl support as a client

2015-07-27 Thread Sourabh Chandak (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14643658#comment-14643658 ] Sourabh Chandak commented on KAFKA-1690: Awesome! When will this be integrated to

Re: Review Request 36871: Patch for KAFKA-2381

2015-07-27 Thread Ashish Singh
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36871/ --- (Updated July 28, 2015, 12:56 a.m.) Review request for kafka. Bugs:

[jira] [Commented] (KAFKA-2381) Possible ConcurrentModificationException while unsubscribing from a topic in new consumer

2015-07-27 Thread Ashish K Singh (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14643693#comment-14643693 ] Ashish K Singh commented on KAFKA-2381: --- Created reviewboard

[jira] [Commented] (KAFKA-2275) Add a ListTopics() API to the new consumer

2015-07-27 Thread Ashish K Singh (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14643710#comment-14643710 ] Ashish K Singh commented on KAFKA-2275: --- [~guozhang] I think this is in good shape

[jira] [Commented] (KAFKA-2301) Deprecate ConsumerOffsetChecker

2015-07-27 Thread Ashish K Singh (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14643714#comment-14643714 ] Ashish K Singh commented on KAFKA-2301: --- [~junrao], [~gwenshap] can any of you help

Re: Review Request 36871: Patch for KAFKA-2381

2015-07-27 Thread Jason Gustafson
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36871/#review93213 --- Ouch. Hard to believe this wasn't caught yet.

[jira] [Updated] (KAFKA-2360) The kafka-consumer-perf-test.sh script help information print useless parameters.

2015-07-27 Thread Bo Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bo Wang updated KAFKA-2360: --- Description: Run kafka-consumer-perf-test.sh --help to show help information, but found 3 parameters

[jira] [Created] (KAFKA-2380) Publish Kafka snapshot Maven artifacts

2015-07-27 Thread Stevo Slavic (JIRA)
Stevo Slavic created KAFKA-2380: --- Summary: Publish Kafka snapshot Maven artifacts Key: KAFKA-2380 URL: https://issues.apache.org/jira/browse/KAFKA-2380 Project: Kafka Issue Type: Task

[jira] [Commented] (KAFKA-2381) Possible ConcurrentModificationException while unsubscribing from a topic in new consumer

2015-07-27 Thread Ashish K Singh (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14643843#comment-14643843 ] Ashish K Singh commented on KAFKA-2381: --- Updated reviewboard

Re: Review Request 36871: Patch for KAFKA-2381

2015-07-27 Thread Ashish Singh
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36871/ --- (Updated July 28, 2015, 4:56 a.m.) Review request for kafka. Bugs:

  1   2   >