[jira] [Commented] (KAFKA-982) Logo for Kafka
[ https://issues.apache.org/jira/browse/KAFKA-982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13719316#comment-13719316 ] Esko Suomi commented on KAFKA-982: -- +1 for 301; it looks like queues/buffers in different states plus has a bit of color and abstractness to it that I like. +1 for 298; less abstract but still quite brutal which I think fits Kafka's approach to message queues than other hand-holding systems. Logo for Kafka -- Key: KAFKA-982 URL: https://issues.apache.org/jira/browse/KAFKA-982 Project: Kafka Issue Type: Improvement Reporter: Jay Kreps Attachments: 289.jpeg, 294.jpeg, 296.png, 298.jpeg, 301.png, 302.png We should have a logo for kafka. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-982) Logo for Kafka
[ https://issues.apache.org/jira/browse/KAFKA-982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13719319#comment-13719319 ] Olson,Andrew commented on KAFKA-982: I will be out of the office without access to email until Tuesday, 8/13/2013. For urgent issues please contact Greg Whitsitt. Andrew Olson | Sr. Software Architect | Cerner Corporation | 816.201.3825 | aols...@cerner.com | www.cerner.com CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are intended only for the addressee. The information contained in this message is confidential and may constitute inside or non-public information under international, federal, or state securities laws. Unauthorized forwarding, printing, copying, distribution, or use of such information is strictly prohibited and may be unlawful. If you are not the addressee, please promptly delete this message and notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024. Logo for Kafka -- Key: KAFKA-982 URL: https://issues.apache.org/jira/browse/KAFKA-982 Project: Kafka Issue Type: Improvement Reporter: Jay Kreps Attachments: 289.jpeg, 294.jpeg, 296.png, 298.jpeg, 301.png, 302.png We should have a logo for kafka. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
kafka pull request: Unmap before resizing
GitHub user lizziew opened a pull request: https://github.com/apache/kafka/pull/6 Unmap before resizing While I was studying how MappedByteBuffer works, I saw a sharing runtime exception on Windows. I applied what I learned to generate a patch which uses an internal open JDK API to solve this problem. --- Caused by: java.io.IOException: The requested operation cannot be performed on a file with a user-mapped section open at java.io.RandomAccessFile.setLength(Native Method) at kafka.log.OffsetIndex.liftedTree2$1(OffsetIndex.scala:263) at kafka.log.OffsetIndex.resize(OffsetIndex.scala:262) at kafka.log.OffsetIndex.trimToValidSize(OffsetIndex.scala:247) at kafka.log.Log.rollToOffset(Log.scala:518) at kafka.log.Log.roll(Log.scala:502) at kafka.log.Log.maybeRoll(Log.scala:484) at kafka.log.Log.append(Log.scala:297) You can merge this pull request into a Git repository by running: $ git pull https://github.com/lizziew/kafka 0.8 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/6.patch
Re: Beta2 release?
I don't think it is. when i was logged in there was only access to staging close and release. I tried to login again but can't so not sure if it is because of recent ldap changes and have to request access again or because I have no uploaded release.. was going to deal with that with the INFRA team when doing the next release if it was still a problem. its a beta release so there should be some expectation of some things not working it should not be a huge major deal for folks they did not even have maven before this and at least have some way to go about it... and if we are only a few weeks away from beta2 it might be worth waiting so that we don't have to have a beta3 and the 0.8.0 GA comes sooner and we get to 0.8.1 with the 2.10 support which I think is an important direction and goal for the project (FTW) /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop / On Wed, Jul 24, 2013 at 11:49 AM, David Arthur mum...@gmail.com wrote: In that case, would it be possible to kill the release that's in Maven Central? It's probably better to have nothing out there rather than have something broken. On 7/24/13 11:43 AM, Jay Kreps wrote: I guess the question is how long it would take us to wrap the remaining items for 0.8 final. If that is just a few weeks it might be okay to wait on it... -jay On Wed, Jul 24, 2013 at 7:20 AM, Joe Stein crypt...@gmail.com wrote: What are the thoughts of a beta2 release? The Maven issues are causing some grief for folks not sure if there are other items we want fixed before GA? It would be great to limit just one more release before GA so we can proceed with 0.8.1 off trunk for other patches (like 2.10 Scala support) folks are looking for. /* Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop **/
[jira] [Commented] (KAFKA-987) Avoid checkpointing offsets in Kafka consumer that have not changed since the last commit
[ https://issues.apache.org/jira/browse/KAFKA-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13719723#comment-13719723 ] Jun Rao commented on KAFKA-987: --- 1. The issue on startup is the following. If a consumer starts up from the end of the log and there is no new message coming in, no offset will be checkpointed to ZK. This will affect tools like ConsumerOffsetChecker. 2. During rebalance, a consumer may pick up offsets committed by other consumer instances. If we don't update the offset cache in addPartitionTopicInfo(), we will do an extra unnecessary offset update to ZK. It seems to me that the impact for #1 is bigger than the slight performance impact in #2. Another way to do that is to always force the very first offset (per partition) write to ZK. However, I am not sure if it's worth the complexity. Avoid checkpointing offsets in Kafka consumer that have not changed since the last commit - Key: KAFKA-987 URL: https://issues.apache.org/jira/browse/KAFKA-987 Project: Kafka Issue Type: Bug Affects Versions: 0.8 Reporter: Swapnil Ghike Assignee: Swapnil Ghike Labels: improvement Fix For: 0.8 Attachments: kafka-987.patch, kafka-987-v2.patch We need to fix the Kafka zookeeper consumer to avoid checkpointing offsets that have not changed since the last offset commit. This will help reduce the write load on zookeeper. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: kafka pull request: Unmap before resizing
Hi, can you file a JIRA and attach the patch there? This does the Apache copyright stuff... Instructions here: http://kafka.apache.org/contributing.html This seems like a good thing to have. Is there a more portable way to do this? Obviously this would not work on a non-sun JVM. I think the ideal would be to add a tryUnmap() method that attempts the case but swallows any exception. Also I would like to fully understand the Sun's objection to allowing user code to unmap buffers. In that ticket on the previous thread on Windows+mmap there was a mention that forcing the unmap was somehow a security hole. I don't understand this. Unix obviously supports unmap and there is no security issue with that as normal process-level memory isolation protects processes from reading each others memory or files. I suspect maybe this would only be true in cases where you are trying to isolate different code in the same JVM which is not a valid use case. But I want to fully understand the issue before we add something someone claims is unsafe. -Jay On Thu, Jul 25, 2013 at 12:07 AM, lizziew g...@git.apache.org wrote: GitHub user lizziew opened a pull request: https://github.com/apache/kafka/pull/6 Unmap before resizing While I was studying how MappedByteBuffer works, I saw a sharing runtime exception on Windows. I applied what I learned to generate a patch which uses an internal open JDK API to solve this problem. --- Caused by: java.io.IOException: The requested operation cannot be performed on a file with a user-mapped section open at java.io.RandomAccessFile.setLength(Native Method) at kafka.log.OffsetIndex.liftedTree2$1(OffsetIndex.scala:263) at kafka.log.OffsetIndex.resize(OffsetIndex.scala:262) at kafka.log.OffsetIndex.trimToValidSize(OffsetIndex.scala:247) at kafka.log.Log.rollToOffset(Log.scala:518) at kafka.log.Log.roll(Log.scala:502) at kafka.log.Log.maybeRoll(Log.scala:484) at kafka.log.Log.append(Log.scala:297) You can merge this pull request into a Git repository by running: $ git pull https://github.com/lizziew/kafka 0.8 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/6.patch
[jira] [Updated] (KAFKA-959) DefaultEventHandler can send more produce requests than necesary
[ https://issues.apache.org/jira/browse/KAFKA-959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-959: -- Resolution: Fixed Fix Version/s: 0.8 Status: Resolved (was: Patch Available) Thanks for the patch. +1 Committed to 0.8. DefaultEventHandler can send more produce requests than necesary Key: KAFKA-959 URL: https://issues.apache.org/jira/browse/KAFKA-959 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8 Reporter: Jun Rao Assignee: Guozhang Wang Fix For: 0.8 Attachments: KAFKA-959.v1.patch In DefaultEventHandler, for a batch of messages, it picks a random partition per message (when there is no key specified). This means that it can send up to P produce requests where P is the number of partitions in a topic. A better way is probably to pick a single random partition for the whole batch of messages. This will reduce the number of produce requests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
kafka pull request: Modified the async producer so it re-queues failed batc...
GitHub user hiloboy0119 opened a pull request: https://github.com/apache/kafka/pull/7 Modified the async producer so it re-queues failed batches. I'm working on an application that needs the throughput offered by an async producer but also needs to handle send failures gracefully like a sync producer. I modified the ProducerSendThread so it will re-queue failed batches. This allowed me to determine the behavior I wanted from my producer with the queue config parameters. A queue.enqueue.timeout.ms=0 allowed me to get runtime exceptions when sends failed often enough to fill the queue. This also allowed me to use queue.buffering.max.messages to control how tolerant the application is to network blips. You can merge this pull request into a Git repository by running: $ git pull https://github.com/gradientx/kafka async_requeue Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/7.patch
[jira] [Updated] (KAFKA-955) After a leader change, messages sent with ack=0 are lost
[ https://issues.apache.org/jira/browse/KAFKA-955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-955: Assignee: Guozhang Wang After a leader change, messages sent with ack=0 are lost Key: KAFKA-955 URL: https://issues.apache.org/jira/browse/KAFKA-955 Project: Kafka Issue Type: Bug Reporter: Jason Rosenberg Assignee: Guozhang Wang If the leader changes for a partition, and a producer is sending messages with ack=0, then messages will be lost, since the producer has no active way of knowing that the leader has changed, until it's next metadata refresh update. The broker receiving the message, which is no longer the leader, logs a message like this: Produce request with correlation id 7136261 from client on partition [mytopic,0] failed due to Leader not local for partition [mytopic,0] on broker 508818741 This is exacerbated by the controlled shutdown mechanism, which forces an immediate leader change. A possible solution to this would be for a broker which receives a message, for a topic that it is no longer the leader for (and if the ack level is 0), then the broker could just silently forward the message over to the current leader. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-955) After a leader change, messages sent with ack=0 are lost
[ https://issues.apache.org/jira/browse/KAFKA-955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-955: Attachment: KAFKA-955.v1.patch After a leader change, messages sent with ack=0 are lost Key: KAFKA-955 URL: https://issues.apache.org/jira/browse/KAFKA-955 Project: Kafka Issue Type: Bug Reporter: Jason Rosenberg Assignee: Guozhang Wang Attachments: KAFKA-955.v1.patch If the leader changes for a partition, and a producer is sending messages with ack=0, then messages will be lost, since the producer has no active way of knowing that the leader has changed, until it's next metadata refresh update. The broker receiving the message, which is no longer the leader, logs a message like this: Produce request with correlation id 7136261 from client on partition [mytopic,0] failed due to Leader not local for partition [mytopic,0] on broker 508818741 This is exacerbated by the controlled shutdown mechanism, which forces an immediate leader change. A possible solution to this would be for a broker which receives a message, for a topic that it is no longer the leader for (and if the ack level is 0), then the broker could just silently forward the message over to the current leader. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-955) After a leader change, messages sent with ack=0 are lost
[ https://issues.apache.org/jira/browse/KAFKA-955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-955: Status: Patch Available (was: Open) Following the close-socket approach, I propose the following change: 1. Add a closeSocket: Boolean field in Response class. 2. In KafkaApi.handleProducerRequest, if requireAck == 0 check if numPartitionsInError != 0. If yes set closeSocket to true in the returning response. 3. SocketServer.Processor.processNewResponses, if curr.responseSend == null, check if closeSocket == true. If yes, log the closing socket info and close the key. After a leader change, messages sent with ack=0 are lost Key: KAFKA-955 URL: https://issues.apache.org/jira/browse/KAFKA-955 Project: Kafka Issue Type: Bug Reporter: Jason Rosenberg Assignee: Guozhang Wang Attachments: KAFKA-955.v1.patch If the leader changes for a partition, and a producer is sending messages with ack=0, then messages will be lost, since the producer has no active way of knowing that the leader has changed, until it's next metadata refresh update. The broker receiving the message, which is no longer the leader, logs a message like this: Produce request with correlation id 7136261 from client on partition [mytopic,0] failed due to Leader not local for partition [mytopic,0] on broker 508818741 This is exacerbated by the controlled shutdown mechanism, which forces an immediate leader change. A possible solution to this would be for a broker which receives a message, for a topic that it is no longer the leader for (and if the ack level is 0), then the broker could just silently forward the message over to the current leader. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-955) After a leader change, messages sent with ack=0 are lost
[ https://issues.apache.org/jira/browse/KAFKA-955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-955: Attachment: KAFKA-955.v1.patch Add one case for ack=0 in testSendWithDeadBroker, passed. After a leader change, messages sent with ack=0 are lost Key: KAFKA-955 URL: https://issues.apache.org/jira/browse/KAFKA-955 Project: Kafka Issue Type: Bug Reporter: Jason Rosenberg Assignee: Guozhang Wang Attachments: KAFKA-955.v1.patch, KAFKA-955.v1.patch If the leader changes for a partition, and a producer is sending messages with ack=0, then messages will be lost, since the producer has no active way of knowing that the leader has changed, until it's next metadata refresh update. The broker receiving the message, which is no longer the leader, logs a message like this: Produce request with correlation id 7136261 from client on partition [mytopic,0] failed due to Leader not local for partition [mytopic,0] on broker 508818741 This is exacerbated by the controlled shutdown mechanism, which forces an immediate leader change. A possible solution to this would be for a broker which receives a message, for a topic that it is no longer the leader for (and if the ack level is 0), then the broker could just silently forward the message over to the current leader. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
