Re: Review Request 14675: address previous review comments
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/14675/ --- (Updated Oct. 17, 2013, 4:33 p.m.) Review request for kafka. Summary (updated) - address previous review comments Bugs: KAFKA-1090 https://issues.apache.org/jira/browse/KAFKA-1090 Repository: kafka Description (updated) --- kafka-1090, delta kafka-1090 Diffs (updated) - core/src/test/scala/unit/kafka/network/SocketServerTest.scala 94b5a2a36b2b82377576c2479a91e6799ae2e326 Diff: https://reviews.apache.org/r/14675/diff/ Testing --- Thanks, Jun Rao
[jira] [Commented] (KAFKA-1090) testPipelinedRequestOrdering has transient failures
[ https://issues.apache.org/jira/browse/KAFKA-1090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798049#comment-13798049 ] Jun Rao commented on KAFKA-1090: Updated reviewboard https://reviews.apache.org/r/14675/ testPipelinedRequestOrdering has transient failures --- Key: KAFKA-1090 URL: https://issues.apache.org/jira/browse/KAFKA-1090 Project: Kafka Issue Type: Bug Components: core Reporter: Jun Rao Assignee: Jun Rao Priority: Minor Fix For: 0.8.1 Attachments: KAFKA-1090_2013-10-17_09:33:17.patch, KAFKA-1090.patch The issue is that after the 1st response is added to the response queue, the socket key may or may not be readable depending on how quickly the response is sent through socket. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (KAFKA-1090) testPipelinedRequestOrdering has transient failures
[ https://issues.apache.org/jira/browse/KAFKA-1090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-1090: --- Attachment: KAFKA-1090_2013-10-17_09:33:17.patch testPipelinedRequestOrdering has transient failures --- Key: KAFKA-1090 URL: https://issues.apache.org/jira/browse/KAFKA-1090 Project: Kafka Issue Type: Bug Components: core Reporter: Jun Rao Assignee: Jun Rao Priority: Minor Fix For: 0.8.1 Attachments: KAFKA-1090_2013-10-17_09:33:17.patch, KAFKA-1090.patch The issue is that after the 1st response is added to the response queue, the socket key may or may not be readable depending on how quickly the response is sent through socket. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Closed] (KAFKA-1091) full topic list can be read from metadata cache in the broker instead of ZK
[ https://issues.apache.org/jira/browse/KAFKA-1091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao closed KAFKA-1091. -- full topic list can be read from metadata cache in the broker instead of ZK --- Key: KAFKA-1091 URL: https://issues.apache.org/jira/browse/KAFKA-1091 Project: Kafka Issue Type: Bug Reporter: Jun Rao Assignee: Jun Rao Priority: Minor Fix For: 0.8.1 Attachments: KAFKA-1091.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Resolved] (KAFKA-1091) full topic list can be read from metadata cache in the broker instead of ZK
[ https://issues.apache.org/jira/browse/KAFKA-1091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-1091. Resolution: Fixed Thanks for the review. Committed to trunk. full topic list can be read from metadata cache in the broker instead of ZK --- Key: KAFKA-1091 URL: https://issues.apache.org/jira/browse/KAFKA-1091 Project: Kafka Issue Type: Bug Reporter: Jun Rao Assignee: Jun Rao Priority: Minor Fix For: 0.8.1 Attachments: KAFKA-1091.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
Re: Review Request 14675: address previous review comments
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/14675/#review27132 --- Ship it! Ship It! - Neha Narkhede On Oct. 17, 2013, 4:33 p.m., Jun Rao wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/14675/ --- (Updated Oct. 17, 2013, 4:33 p.m.) Review request for kafka. Bugs: KAFKA-1090 https://issues.apache.org/jira/browse/KAFKA-1090 Repository: kafka Description --- kafka-1090, delta kafka-1090 Diffs - core/src/test/scala/unit/kafka/network/SocketServerTest.scala 94b5a2a36b2b82377576c2479a91e6799ae2e326 Diff: https://reviews.apache.org/r/14675/diff/ Testing --- Thanks, Jun Rao
[jira] [Updated] (KAFKA-1090) testPipelinedRequestOrdering has transient failures
[ https://issues.apache.org/jira/browse/KAFKA-1090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-1090: --- Resolution: Fixed Status: Resolved (was: Patch Available) Thanks for the review. Committed to trunk after changing the test case to a better name. testPipelinedRequestOrdering has transient failures --- Key: KAFKA-1090 URL: https://issues.apache.org/jira/browse/KAFKA-1090 Project: Kafka Issue Type: Bug Components: core Reporter: Jun Rao Assignee: Jun Rao Priority: Minor Fix For: 0.8.1 Attachments: KAFKA-1090_2013-10-17_09:33:17.patch, KAFKA-1090.patch The issue is that after the 1st response is added to the response queue, the socket key may or may not be readable depending on how quickly the response is sent through socket. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Closed] (KAFKA-1090) testPipelinedRequestOrdering has transient failures
[ https://issues.apache.org/jira/browse/KAFKA-1090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao closed KAFKA-1090. -- testPipelinedRequestOrdering has transient failures --- Key: KAFKA-1090 URL: https://issues.apache.org/jira/browse/KAFKA-1090 Project: Kafka Issue Type: Bug Components: core Reporter: Jun Rao Assignee: Jun Rao Priority: Minor Fix For: 0.8.1 Attachments: KAFKA-1090_2013-10-17_09:33:17.patch, KAFKA-1090.patch The issue is that after the 1st response is added to the response queue, the socket key may or may not be readable depending on how quickly the response is sent through socket. -- This message was sent by Atlassian JIRA (v6.1#6144)
New Committer
I am pleased to announce that David Arthur has been voted to become a committer for Apache Kafka. Welcome, David, and thanks for your ongoing contributions! /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop /
[jira] Subscription: outstanding kafka patches
Issue Subscription Filter: outstanding kafka patches (65 issues) The list of outstanding kafka patches Subscriber: kafka-mailing-list Key Summary KAFKA-1086 Improve GetOffsetShell to find metadata automatically https://issues.apache.org/jira/browse/KAFKA-1086 KAFKA-1082 zkclient dies after UnknownHostException in zk reconnect https://issues.apache.org/jira/browse/KAFKA-1082 KAFKA-1049 Encoder implementations are required to provide an undocumented constructor. https://issues.apache.org/jira/browse/KAFKA-1049 KAFKA-1042 Fix segment flush logic https://issues.apache.org/jira/browse/KAFKA-1042 KAFKA-1032 Messages sent to the old leader will be lost on broker GC resulted failure https://issues.apache.org/jira/browse/KAFKA-1032 KAFKA-1020 Remove getAllReplicasOnBroker from KafkaController https://issues.apache.org/jira/browse/KAFKA-1020 KAFKA-1012 Implement an Offset Manager and hook offset requests to it https://issues.apache.org/jira/browse/KAFKA-1012 KAFKA-1011 Decompression and re-compression on MirrorMaker could result in messages being dropped in the pipeline https://issues.apache.org/jira/browse/KAFKA-1011 KAFKA-1005 kafka.perf.ConsumerPerformance not shutting down consumer https://issues.apache.org/jira/browse/KAFKA-1005 KAFKA-1004 Handle topic event for trivial whitelist topic filters https://issues.apache.org/jira/browse/KAFKA-1004 KAFKA-998 Producer should not retry on non-recoverable error codes https://issues.apache.org/jira/browse/KAFKA-998 KAFKA-997 Provide a strict verification mode when reading configuration properties https://issues.apache.org/jira/browse/KAFKA-997 KAFKA-996 Capitalize first letter for log entries https://issues.apache.org/jira/browse/KAFKA-996 KAFKA-984 Avoid a full rebalance in cases when a new topic is discovered but container/broker set stay the same https://issues.apache.org/jira/browse/KAFKA-984 KAFKA-976 Order-Preserving Mirror Maker Testcase https://issues.apache.org/jira/browse/KAFKA-976 KAFKA-967 Use key range in ProducerPerformance https://issues.apache.org/jira/browse/KAFKA-967 KAFKA-917 Expose zk.session.timeout.ms in console consumer https://issues.apache.org/jira/browse/KAFKA-917 KAFKA-885 sbt package builds two kafka jars https://issues.apache.org/jira/browse/KAFKA-885 KAFKA-881 Kafka broker not respecting log.roll.hours https://issues.apache.org/jira/browse/KAFKA-881 KAFKA-873 Consider replacing zkclient with curator (with zkclient-bridge) https://issues.apache.org/jira/browse/KAFKA-873 KAFKA-868 System Test - add test case for rolling controlled shutdown https://issues.apache.org/jira/browse/KAFKA-868 KAFKA-863 System Test - update 0.7 version of kafka-run-class.sh for Migration Tool test cases https://issues.apache.org/jira/browse/KAFKA-863 KAFKA-859 support basic auth protection of mx4j console https://issues.apache.org/jira/browse/KAFKA-859 KAFKA-855 Ant+Ivy build for Kafka https://issues.apache.org/jira/browse/KAFKA-855 KAFKA-854 Upgrade dependencies for 0.8 https://issues.apache.org/jira/browse/KAFKA-854 KAFKA-815 Improve SimpleConsumerShell to take in a max messages config option https://issues.apache.org/jira/browse/KAFKA-815 KAFKA-745 Remove getShutdownReceive() and other kafka specific code from the RequestChannel https://issues.apache.org/jira/browse/KAFKA-745 KAFKA-735 Add looping and JSON output for ConsumerOffsetChecker https://issues.apache.org/jira/browse/KAFKA-735 KAFKA-717 scala 2.10 build support https://issues.apache.org/jira/browse/KAFKA-717 KAFKA-686 0.8 Kafka broker should give a better error message when running against 0.7 zookeeper https://issues.apache.org/jira/browse/KAFKA-686 KAFKA-674 Clean Shutdown Testing - Log segments checksums mismatch https://issues.apache.org/jira/browse/KAFKA-674 KAFKA-652 Create testcases for clean shut-down https://issues.apache.org/jira/browse/KAFKA-652 KAFKA-649 Cleanup log4j logging https://issues.apache.org/jira/browse/KAFKA-649 KAFKA-645 Create a shell script to run System Test with DEBUG details and tee console output to a file https://issues.apache.org/jira/browse/KAFKA-645 KAFKA-598 decouple fetch size from max message size https://issues.apache.org/jira/browse/KAFKA-598 KAFKA-583 SimpleConsumerShell may receive less data inconsistently https://issues.apache.org/jira/browse/KAFKA-583 KAFKA-559 Garbage collect old consumer metadata entries https://issues.apache.org/jira/browse/KAFKA-559 KAFKA-552 No error messages logged for those failing-to-send messages from Producer
Fwd: Special Bay Area HUG: Tajo and Samza
FYI. -- Forwarded message -- From: Jakob Homan jgho...@gmail.com Date: Thu, Oct 17, 2013 at 11:08 AM Subject: Special Bay Area HUG: Tajo and Samza To: d...@samza.incubator.apache.org Hey everybody- Join us at LinkedIn Nov. 5 for a special HUG dedicated to two new awesome Incubator projects, Tajo, a low-latency SQL query engine atop YARN and Samza. http://www.meetup.com/hadoop/events/146077932/ -Jakob
[jira] [Created] (KAFKA-1092) Add server config parameter to separate bind address and ZK hostname
Roger Hoover created KAFKA-1092: --- Summary: Add server config parameter to separate bind address and ZK hostname Key: KAFKA-1092 URL: https://issues.apache.org/jira/browse/KAFKA-1092 Project: Kafka Issue Type: New Feature Components: config Affects Versions: 0.8.1 Reporter: Roger Hoover Currently, in server.properties, you can configure host.name which gets used for two purposes: 1) to bind the socket 2) to publish the broker details to ZK for clients to use. There are times when these two settings need to be different. Here's an example. I want to setup Kafka brokers on OpenStack virtual machines in a private cloud but I need producers to connect from elsewhere on the internal corporate network. With OpenStack, the virtual machines are only exposed to DHCP addresses (typically RFC 1918 private addresses). You can assign floating ips to a virtual machine but it's forwarded using Network Address Translation and not exposed directly to the VM. Also, there's typically no DNS to provide hostname lookup. Hosts have names like fubar.novalocal that are not externally routable. Here's what I want. I want the broker to bind to the VM's private network IP but I want it to publish it's floating IP to ZooKeeper so that producers can publish to it. I propose a new optional parameter, listen, which would allow you to specify the socket address to listen on. If not set, the parameter would default to host.name, which is the current behavior. #Publish the externally routable IP in ZK host.name = floating ip #Accept connections from any interface the VM knows about listen = * -- This message was sent by Atlassian JIRA (v6.1#6144)
Review Request 14730: Patch for KAFKA-1001
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/14730/ --- Review request for kafka. Bugs: KAFKA-1001 https://issues.apache.org/jira/browse/KAFKA-1001 Repository: kafka Description --- KAFKA-1001.v1.6 KAFKA-1001.v1.5 KAFKA-1001.v1 Diffs - core/src/main/scala/kafka/cluster/Partition.scala 5ccecd179d33abfc14dcefc35dd68de7474c6978 core/src/main/scala/kafka/common/ErrorMapping.scala 153bc0b078d21200c02c47dd5ad9b7a7e3326ec4 core/src/main/scala/kafka/consumer/ConsumerFetcherManager.scala 566ca46d113ee7da4b38ee57302ba183b59ab5d6 core/src/main/scala/kafka/consumer/ConsumerFetcherThread.scala dda0a8f041f242bf8a501a8cbd2b9c0258323f96 core/src/main/scala/kafka/log/LogManager.scala 47197153c5d3797d2e2a2f9539d9cd55501468e3 core/src/main/scala/kafka/server/AbstractFetcherManager.scala 15b7bd31446ffb97b8ed0fa6461649a01d81c7e9 core/src/main/scala/kafka/server/AbstractFetcherThread.scala c64260f12bdd6b6c964875e1f3873156442e44e1 core/src/main/scala/kafka/server/ReplicaManager.scala ee1cc0cf451b691eb91d9158ca765aeb60fc3dc8 Diff: https://reviews.apache.org/r/14730/diff/ Testing --- Thanks, Guozhang Wang
[jira] [Updated] (KAFKA-1001) Handle follower transition in batch
[ https://issues.apache.org/jira/browse/KAFKA-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-1001: - Attachment: KAFKA-1001.patch Handle follower transition in batch --- Key: KAFKA-1001 URL: https://issues.apache.org/jira/browse/KAFKA-1001 Project: Kafka Issue Type: Improvement Reporter: Jay Kreps Assignee: Guozhang Wang Fix For: 0.8.1 Attachments: KAFKA-1001.patch In KAFKA-615 we made changes to avoid fsync'ing the active segment of the log due to log roll and maintaining recovery semantics. One downside of the fix for that issue was that it required checkpointing the recovery point for the log many times, one for each partition that transitioned to follower state. In this ticket I aim to fix that issue by making the following changes: 1. Add a new API LogManager.truncateTo(m: Map[TopicAndPartition, Long]). This method will first checkpoint the recovery point, then truncate each of the given logs to the given offset. This method will have to ensure these two things happen atomically. 2. Change ReplicaManager to first stop fetching for all partitions changing to follower state, then call LogManager.truncateTo then complete the existing logic. We think this will, over all, be a good thing. The reason is that the fetching thread current does something like (a) acquire lock, (b) fetch partitions, (c) write data to logs, (d) release locks. Since we currently remove fetchers one at a time this requires acquiring the fetcher lock, and hence generally blocking for half of the read/write cycle for each partition. By doing this in bulk we will avoid reacquiring the lock over and over for each change. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (KAFKA-1001) Handle follower transition in batch
[ https://issues.apache.org/jira/browse/KAFKA-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13798721#comment-13798721 ] Guozhang Wang commented on KAFKA-1001: -- Created reviewboard https://reviews.apache.org/r/14730/ Handle follower transition in batch --- Key: KAFKA-1001 URL: https://issues.apache.org/jira/browse/KAFKA-1001 Project: Kafka Issue Type: Improvement Reporter: Jay Kreps Assignee: Guozhang Wang Fix For: 0.8.1 Attachments: KAFKA-1001.patch In KAFKA-615 we made changes to avoid fsync'ing the active segment of the log due to log roll and maintaining recovery semantics. One downside of the fix for that issue was that it required checkpointing the recovery point for the log many times, one for each partition that transitioned to follower state. In this ticket I aim to fix that issue by making the following changes: 1. Add a new API LogManager.truncateTo(m: Map[TopicAndPartition, Long]). This method will first checkpoint the recovery point, then truncate each of the given logs to the given offset. This method will have to ensure these two things happen atomically. 2. Change ReplicaManager to first stop fetching for all partitions changing to follower state, then call LogManager.truncateTo then complete the existing logic. We think this will, over all, be a good thing. The reason is that the fetching thread current does something like (a) acquire lock, (b) fetch partitions, (c) write data to logs, (d) release locks. Since we currently remove fetchers one at a time this requires acquiring the fetcher lock, and hence generally blocking for half of the read/write cycle for each partition. By doing this in bulk we will avoid reacquiring the lock over and over for each change. -- This message was sent by Atlassian JIRA (v6.1#6144)
