[jira] [Updated] (KAFKA-1841) OffsetCommitRequest API - timestamp field is not versioned

2015-01-05 Thread Joe Stein (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein updated KAFKA-1841: - Fix Version/s: 0.8.2 OffsetCommitRequest API - timestamp field is not versioned

[jira] [Updated] (KAFKA-1012) Implement an Offset Manager and hook offset requests to it

2015-01-05 Thread Joe Stein (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein updated KAFKA-1012: - Fix Version/s: 0.8.2 Implement an Offset Manager and hook offset requests to it

[jira] [Commented] (KAFKA-1841) OffsetCommitRequest API - timestamp field is not versioned

2015-01-05 Thread Joe Stein (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265798#comment-14265798 ] Joe Stein commented on KAFKA-1841: -- In addition to the issue you bring up, the

[jira] [Commented] (KAFKA-1499) Broker-side compression configuration

2015-01-05 Thread Manikumar Reddy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14264537#comment-14264537 ] Manikumar Reddy commented on KAFKA-1499: [~jjkoshy] Uploaded a new patch with some

Re: Optimistic locking

2015-01-05 Thread Daniel Schierbeck
Regarding the use case – some events may only be valid given a specific state of the world, e.g. a NewComment event and a PostClosedForComments event would be valid only in that order. If two producer processes (e.g. two HTTP application processes) tries to write an event each, you may get

Re: Optimistic locking

2015-01-05 Thread Colin Clark
Hi, Couple of comments on this. What you're proposing is difficult to do at scale and would require some type of Paxos style algorithm for the update only if different - it would be easier in that case to just go ahead and do the update. Also, it seems like a conflation of concerns - in an

Re: Optimistic locking

2015-01-05 Thread Daniel Schierbeck
Reading your reply again, I'd like to address this: Also, it seems like a conflation of concerns - in an event sourcing model, we save the immutable event, and represent current state in another, separate data structure. That's the idea – events are stored in Kafka and processed sequentially by

Re: Optimistic locking

2015-01-05 Thread Daniel Schierbeck
Reading the source code, it seems that kafka.log.LogCleaner builds a kafka.log.OffsetMap only when compacting logs, not ahead of time, so the information is not available at write time :-/ If I'm not mistaken, this also means that the cleaner needs _two_ passes to clean a log segment, one to

[jira] [Updated] (KAFKA-1512) Limit the maximum number of connections per ip address

2015-01-05 Thread Jeff Holoman (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Holoman updated KAFKA-1512: Attachment: KAFKA-1512-082.patch Patch against 0.8.2 Limit the maximum number of connections per

[jira] [Commented] (KAFKA-1785) Consumer offset checker should show the offset manager and offsets partition

2015-01-05 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265205#comment-14265205 ] Neha Narkhede commented on KAFKA-1785: -- Ping. [~jjkoshy], [~mgharat] Consumer

[jira] [Commented] (KAFKA-1013) Modify existing tools as per the changes in KAFKA-1000

2015-01-05 Thread Mayuresh Gharat (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265225#comment-14265225 ] Mayuresh Gharat commented on KAFKA-1013: Yes. I suppose the code will require a

Re: Latency Tracking Across All Kafka Component

2015-01-05 Thread Otis Gospodnetic
Hi, That sounds a bit like needing a full, cross-app, cross-network transaction/call tracing, and not something specific or limited to Kafka, doesn't it? Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/ On Mon,

Re: Review Request 29210: Patch for KAFKA-1819

2015-01-05 Thread Neha Narkhede
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29210/#review66733 --- core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala

[jira] [Updated] (KAFKA-1784) Implement a ConsumerOffsetClient library

2015-01-05 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1784: - Resolution: Duplicate Status: Resolved (was: Patch Available) Closing as duplicate of

[jira] [Commented] (KAFKA-1819) Cleaner gets confused about deleted and re-created topics

2015-01-05 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265187#comment-14265187 ] Neha Narkhede commented on KAFKA-1819: -- [~gwenshap]. Thanks for incorporating the

[jira] [Commented] (KAFKA-1731) add config/jmx changes in 0.8.2 doc

2015-01-05 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265203#comment-14265203 ] Neha Narkhede commented on KAFKA-1731: -- +1 on this patch. If there are no more

[jira] [Commented] (KAFKA-1013) Modify existing tools as per the changes in KAFKA-1000

2015-01-05 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265214#comment-14265214 ] Neha Narkhede commented on KAFKA-1013: -- [~mgharat], [~jjkoshy], Are you guys hoping

Re: Review Request 28769: Patch for KAFKA-1809

2015-01-05 Thread Gwen Shapira
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28769/ --- (Updated Jan. 5, 2015, 10:24 p.m.) Review request for kafka. Bugs:

[jira] [Commented] (KAFKA-1809) Refactor brokers to allow listening on multiple ports and IPs

2015-01-05 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265232#comment-14265232 ] Gwen Shapira commented on KAFKA-1809: - Updated reviewboard

[jira] [Updated] (KAFKA-1809) Refactor brokers to allow listening on multiple ports and IPs

2015-01-05 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-1809: Attachment: KAFKA-1809_2015-01-05_14:23:57.patch Refactor brokers to allow listening on multiple

Re: Review Request 29590: 1. Removed defaults for serializer/deserializer. 2. Converted type cast exception to serialization exception in the producer. 3. Added string ser/deser. 4. Moved the isKey fl

2015-01-05 Thread Jay Kreps
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29590/#review66773 ---

[jira] [Commented] (KAFKA-1785) Consumer offset checker should show the offset manager and offsets partition

2015-01-05 Thread Mayuresh Gharat (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265465#comment-14265465 ] Mayuresh Gharat commented on KAFKA-1785: Talked to Joel about it last time. Joel

Re: Review Request 29590: 1. Removed defaults for serializer/deserializer. 2. Converted type cast exception to serialization exception in the producer. 3. Added string ser/deser. 4. Moved the isKey fl

2015-01-05 Thread Jay Kreps
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29590/#review66776 --- Ship it! I'm +1 with those minor comments addressed as you see

[jira] [Commented] (KAFKA-1819) Cleaner gets confused about deleted and re-created topics

2015-01-05 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265503#comment-14265503 ] Gwen Shapira commented on KAFKA-1819: - Moving the check will not always verify

[jira] [Commented] (KAFKA-1809) Refactor brokers to allow listening on multiple ports and IPs

2015-01-05 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265717#comment-14265717 ] Gwen Shapira commented on KAFKA-1809: - Upgrade tests pass, system tests pass, and I've

[jira] [Updated] (KAFKA-1809) Refactor brokers to allow listening on multiple ports and IPs

2015-01-05 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-1809: Attachment: KAFKA-1809_2015-01-05_20:25:15.patch Refactor brokers to allow listening on multiple

Re: Review Request 28769: Patch for KAFKA-1809

2015-01-05 Thread Gwen Shapira
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28769/ --- (Updated Jan. 6, 2015, 4:25 a.m.) Review request for kafka. Bugs: KAFKA-1809

[jira] [Commented] (KAFKA-1809) Refactor brokers to allow listening on multiple ports and IPs

2015-01-05 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265686#comment-14265686 ] Gwen Shapira commented on KAFKA-1809: - Updated reviewboard

Re: Review Request 28769: Patch for KAFKA-1809

2015-01-05 Thread Gwen Shapira
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28769/ --- (Updated Jan. 6, 2015, 5:40 a.m.) Review request for kafka. Bugs: KAFKA-1809

[jira] [Commented] (KAFKA-1809) Refactor brokers to allow listening on multiple ports and IPs

2015-01-05 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265715#comment-14265715 ] Gwen Shapira commented on KAFKA-1809: - Updated reviewboard

[jira] [Updated] (KAFKA-1809) Refactor brokers to allow listening on multiple ports and IPs

2015-01-05 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-1809: Attachment: KAFKA-1809_2015-01-05_21:40:14.patch Refactor brokers to allow listening on multiple

[jira] [Updated] (KAFKA-1642) [Java New Producer Kafka Trunk] CPU Usage Spike to 100% when network connection is lost

2015-01-05 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-1642: - Attachment: KAFKA-1642_2015-01-05_18:56:55.patch [Java New Producer Kafka Trunk]

[jira] [Commented] (KAFKA-1642) [Java New Producer Kafka Trunk] CPU Usage Spike to 100% when network connection is lost

2015-01-05 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265602#comment-14265602 ] Ewen Cheslack-Postava commented on KAFKA-1642: -- Updated reviewboard

Re: Review Request 28582: Patch for KAFKA-1642

2015-01-05 Thread Ewen Cheslack-Postava
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28582/ --- (Updated Jan. 6, 2015, 2:56 a.m.) Review request for kafka. Bugs: KAFKA-1642

Re: Follow-up On Important Issues for 0.8.2

2015-01-05 Thread Bhavesh Mistry
Hi Kafka Dev Team, We have to similar change for # 2 issue, I would like to know if /KAFKA-1788 https://issues.apache.org/jira/browse/KAFKA-1788 will be part of release. Secondly, issue #1, I would like to know what is expected behavior of metrics error counting and call back in (async mode)

[jira] [Commented] (KAFKA-1661) Move MockConsumer and MockProducer from src/main to src/test

2015-01-05 Thread jaikiran pai (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265649#comment-14265649 ] jaikiran pai commented on KAFKA-1661: - I think it's a matter of how Kafka wants to

[jira] [Comment Edited] (KAFKA-1661) Move MockConsumer and MockProducer from src/main to src/test

2015-01-05 Thread jaikiran pai (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265649#comment-14265649 ] jaikiran pai edited comment on KAFKA-1661 at 1/6/15 3:45 AM: -

Re: Review Request 29379: Patch for KAFKA-1788

2015-01-05 Thread Ewen Cheslack-Postava
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29379/#review66709 --- I think the basic approach in this patch looks sound and should

[jira] [Updated] (KAFKA-1841) OffsetCommitRequest API - timestamp field is not versioned

2015-01-05 Thread Dana Powers (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dana Powers updated KAFKA-1841: --- Description: Timestamp field was added to the OffsetCommitRequest wire protocol api for 0.8.2 by

[jira] [Commented] (KAFKA-1634) Improve semantics of timestamp in OffsetCommitRequests and update documentation

2015-01-05 Thread Dana Powers (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265042#comment-14265042 ] Dana Powers commented on KAFKA-1634: possibly related to this JIRA: KAFKA-1841 . The

[jira] [Commented] (KAFKA-1661) Move MockConsumer and MockProducer from src/main to src/test

2015-01-05 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14264860#comment-14264860 ] Jay Kreps commented on KAFKA-1661: -- That would work, but is examples really better? I

[jira] [Commented] (KAFKA-1723) make the metrics name in new producer more standard

2015-01-05 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14264870#comment-14264870 ] Jay Kreps commented on KAFKA-1723: -- Makes sense. I'd like to keep the metric data model

Re: Review Request 29468: Patch for KAFKA-1805

2015-01-05 Thread Ewen Cheslack-Postava
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29468/#review66688 ---

[jira] [Comment Edited] (KAFKA-1809) Refactor brokers to allow listening on multiple ports and IPs

2015-01-05 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14264127#comment-14264127 ] Gwen Shapira edited comment on KAFKA-1809 at 1/5/15 7:21 PM: -

[jira] [Comment Edited] (KAFKA-1809) Refactor brokers to allow listening on multiple ports and IPs

2015-01-05 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14264127#comment-14264127 ] Gwen Shapira edited comment on KAFKA-1809 at 1/5/15 7:23 PM: -

Re: Review Request 29072: Patch for kafka-1797

2015-01-05 Thread Jun Rao
On Jan. 2, 2015, 5:34 p.m., Jay Kreps wrote: I'm a little late to this party but I think there are a couple of issues we should address before the release: 1. Currently the producer defaults to org.apache.kafka.clients.consumer.ByteArrayDeserializer. It should not take a default.

Latency Tracking Across All Kafka Component

2015-01-05 Thread Bhavesh Mistry
Hi Kafka Team/Users, We are using Linked-in Kafka data pipe-line end-to-end. Producer(s) -Local DC Brokers - MM - Central brokers - Camus Job - HDFS This is working out very well for us, but we need to have visibility of latency at each layer (Local DC Brokers - MM - Central brokers - Camus Job

Review Request 29590: 1. Removed defaults for serializer/deserializer. 2. Converted type cast exception to serialization exception in the producer. 3. Added string ser/deser. 4. Moved the isKey flag t

2015-01-05 Thread Jun Rao
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29590/ --- Review request for kafka. Bugs: kafka-1797

[jira] [Updated] (KAFKA-1797) add the serializer/deserializer api to the new java client

2015-01-05 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-1797: --- Attachment: kafka-1797.patch add the serializer/deserializer api to the new java client

[jira] [Commented] (KAFKA-1797) add the serializer/deserializer api to the new java client

2015-01-05 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14264990#comment-14264990 ] Jun Rao commented on KAFKA-1797: Created reviewboard

[jira] [Commented] (KAFKA-1512) Limit the maximum number of connections per ip address

2015-01-05 Thread Joe Stein (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14264667#comment-14264667 ] Joe Stein commented on KAFKA-1512: -- I was just about to commit this and realized this is

[jira] [Updated] (KAFKA-1512) Limit the maximum number of connections per ip address

2015-01-05 Thread Joe Stein (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein updated KAFKA-1512: - Affects Version/s: 0.8.2 Limit the maximum number of connections per ip address