[jira] [Commented] (KAFKA-982) Logo for Kafka
[ https://issues.apache.org/jira/browse/KAFKA-982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13718029#comment-13718029 ] Johan Lundahl commented on KAFKA-982: - +1 for 298, 301 Logo for Kafka -- Key: KAFKA-982 URL: https://issues.apache.org/jira/browse/KAFKA-982 Project: Kafka Issue Type: Improvement Reporter: Jay Kreps Attachments: 289.jpeg, 294.jpeg, 296.png, 298.jpeg, 301.png We should have a logo for kafka. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Does Kafka message manager tool (read messages from console, delete messages) exist?
Hi Jun Rao and thanks for your answer. Re 1: Will try to do it way you have described. As I understood getOffsetBefore returns offset for set of compressed messages. In my case it will mean that my from=timestamp_1 and to=timestamp_2 will work roughly (I will have to double check timestamp for each message inside tool and decide fit this message my criteria or not), right? Is there any chance to get API with honest offset for particular time? Re 2: Can I use https://issues.apache.org/jira/browse/KAFKA-554 for configuring topic.log.roll.hours on the fly? Should I restart Kafka after changes in config by KAFKA-554https://issues.apache.org/jira/browse/KAFKA-554? How about to=timestamp_2. With topic.log.roll.hours I can define only one border for time range. What about second (right border on timeline)? Thanks, Vladimir. On Wed, Jul 24, 2013 at 7:42 AM, Jun Rao jun...@gmail.com wrote: For 1, you can write a tool that uses our getOffsetBefore api and SimpleConsumer. The offset returned is at the log segment boundary. So it's not going to match the specified time precisely. For 2, you can configure a topic to be rolled by timestamp (log.roll.hours and topic.log.roll.hours). Then set the retention time accordingly. Thanks, Jun On Tue, Jul 23, 2013 at 5:21 AM, Vladimir Tretyakov vladimir.tretya...@sematext.com wrote: Hi, we use Kafka (0.7.2 version) in our product as message delivery service and message storage (our retention time 48h) system. Thanks a lot for perfect tool/lib. Now we really need 2 additional features: 1. Sometimes we need look at our messages we have in Kafka from command line, for example: ~$ kafkaTool get topic=my_topic_name from=timestamp_1 to=timestamp_2 | grep 'Exception' We need this feature because we want to have ability to see our raw messages body. 2. Sometimes we have to delete messages older than given timestamp. Or in other words we have to change offset for all consumers for given topic. (But we don't like to work with offset because we know only timestamp we like to move our consumers to). Example: ~$ kafkaTool purge topic=my_topic_name from=timestamp_1 to=timestamp_2 We need this feature because we want to skip processing of some messages by our regular consumers. Is these 2 features exist? If not, what is the best way to do this using the existing APIs? According to https://issues.apache.org/jira/browse/KAFKA-260 there is plan to add timestamp to message. Is it possible, or is it planned, to have ... something like FetchRequest, but with timestamp as parameter instead of offset? If this is currently not possible, should I open a JIRA issue? Best regards, Vladimir.
[jira] [Commented] (KAFKA-987) Avoid checkpointing offsets in Kafka consumer that have not changed since the last commit
[ https://issues.apache.org/jira/browse/KAFKA-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13718044#comment-13718044 ] Swapnil Ghike commented on KAFKA-987: - I think that any call to createMessageStreams will trigger a rebalance, that will fill up the topicregistry and the checkpointing of offsets will start regardless of whether new messages are being consumed or not. Hence, we should probably update the cached checkpointedOffsets map in addPartitionTopicInfo(). May be I have missed something? Avoid checkpointing offsets in Kafka consumer that have not changed since the last commit - Key: KAFKA-987 URL: https://issues.apache.org/jira/browse/KAFKA-987 Project: Kafka Issue Type: Bug Affects Versions: 0.8 Reporter: Swapnil Ghike Assignee: Swapnil Ghike Labels: improvement Fix For: 0.8 Attachments: kafka-987.patch, kafka-987-v2.patch We need to fix the Kafka zookeeper consumer to avoid checkpointing offsets that have not changed since the last offset commit. This will help reduce the write load on zookeeper. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Meetup in Raleigh/Durham, NC
Here are the slides from the meetup http://www.slideshare.net/mumrah/kafka-talk-tri-hug We had 40-50 people show up which is about average for our meetup. Roughly half of the audience had heard of Kafka before this talk, and about 10 or so folks had used it or were using it in production. On 7/22/13 1:07 AM, Jun Rao wrote: David, Thanks for sharing this. Jun On Thu, Jul 18, 2013 at 9:19 PM, David Arthur mum...@gmail.com wrote: There is a Hadoop meetup happening in Durham next week. I'm presenting an intro to Kafka. I don't suspect there are any Kafka users in the area who are not already members of TriHUG, but sending this email out just in case :) http://www.meetup.com/TriHUG/**events/129831822/http://www.meetup.com/TriHUG/events/129831822/ -David
[jira] [Commented] (KAFKA-981) Unable to pull Kafka binaries with Ivy
[
https://issues.apache.org/jira/browse/KAFKA-981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13718356#comment-13718356
]
David Arthur commented on KAFKA-981:
Bump. We really should get the invalid pom out of Maven Central, it's affecting
a lot of people. I'm not 100% sure, but I think once a release has been
signed/published it is immutable. We probably need to delete beta1 and create
beta2
Thoughts?
Unable to pull Kafka binaries with Ivy
--
Key: KAFKA-981
URL: https://issues.apache.org/jira/browse/KAFKA-981
Project: Kafka
Issue Type: Bug
Components: packaging
Affects Versions: 0.8
Reporter: David Arthur
Assignee: Joe Stein
Attachments: ant.log, ivy.xml
I am trying to pull the published Kafka binary with a simple Ivy file.
dependency org=org.apache.kafka name=kafka_2.9.2 rev=0.8.0-beta1
conf=default-default/
I get the following exception: [ivy:resolve] problem occurred while resolving
dependency: org.apache.kafka#kafka_2.9.2;0.8.0-beta1 {default=[default]} with
main: java.lang.IllegalArgumentException: null name not allowed
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
Beta2 release?
What are the thoughts of a beta2 release? The Maven issues are causing some grief for folks not sure if there are other items we want fixed before GA? It would be great to limit just one more release before GA so we can proceed with 0.8.1 off trunk for other patches (like 2.10 Scala support) folks are looking for. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop /
Re: Beta2 release?
I think fixing the pom is enough reason to do another beta drop. While we're talking about the pom and Maven... Joe, have you given any thought to using Maven classifiers to support the multiple scala versions? E.g., dependency groupIdorg.apache.kafka/groupId artifactIdkafka/artifactId version0.8-beta/version classifierscala-2.9.2/classifier /dependency dependency org=org.apache.kafka name=kafka rev=0.8-beta m:classifier=scala-2.8.0/ val kafka = org.apache.kafka % kafka % 0.8-beta classifier scala-2.8.0 -David On 7/24/13 10:20 AM, Joe Stein wrote: What are the thoughts of a beta2 release? The Maven issues are causing some grief for folks not sure if there are other items we want fixed before GA? It would be great to limit just one more release before GA so we can proceed with 0.8.1 off trunk for other patches (like 2.10 Scala support) folks are looking for. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop /
Re: Beta2 release?
I guess the question is how long it would take us to wrap the remaining items for 0.8 final. If that is just a few weeks it might be okay to wait on it... -jay On Wed, Jul 24, 2013 at 7:20 AM, Joe Stein crypt...@gmail.com wrote: What are the thoughts of a beta2 release? The Maven issues are causing some grief for folks not sure if there are other items we want fixed before GA? It would be great to limit just one more release before GA so we can proceed with 0.8.1 off trunk for other patches (like 2.10 Scala support) folks are looking for. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop /
Re: Beta2 release?
In that case, would it be possible to kill the release that's in Maven Central? It's probably better to have nothing out there rather than have something broken. On 7/24/13 11:43 AM, Jay Kreps wrote: I guess the question is how long it would take us to wrap the remaining items for 0.8 final. If that is just a few weeks it might be okay to wait on it... -jay On Wed, Jul 24, 2013 at 7:20 AM, Joe Stein crypt...@gmail.com wrote: What are the thoughts of a beta2 release? The Maven issues are causing some grief for folks not sure if there are other items we want fixed before GA? It would be great to limit just one more release before GA so we can proceed with 0.8.1 off trunk for other patches (like 2.10 Scala support) folks are looking for. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop /
[jira] [Commented] (KAFKA-982) Logo for Kafka
[ https://issues.apache.org/jira/browse/KAFKA-982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13718510#comment-13718510 ] Jay Kreps commented on KAFKA-982: - +1 296 Logo for Kafka -- Key: KAFKA-982 URL: https://issues.apache.org/jira/browse/KAFKA-982 Project: Kafka Issue Type: Improvement Reporter: Jay Kreps Attachments: 289.jpeg, 294.jpeg, 296.png, 298.jpeg, 301.png We should have a logo for kafka. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-987) Avoid checkpointing offsets in Kafka consumer that have not changed since the last commit
[ https://issues.apache.org/jira/browse/KAFKA-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13718567#comment-13718567 ] Neha Narkhede commented on KAFKA-987: - Jun, I think what you are suggesting makes sense on startup before the consumer has consumed any messages. However, since the offset map is a cache for what's in zookeeper, the safest way is to keep it in sync with the zookeeper data. Before the consumer can pull any data, it has to rebalance and while rebalancing we read the offsets from zk anyways. So I think it is correct to update the offset cache in addPartitionTopicInfo() Avoid checkpointing offsets in Kafka consumer that have not changed since the last commit - Key: KAFKA-987 URL: https://issues.apache.org/jira/browse/KAFKA-987 Project: Kafka Issue Type: Bug Affects Versions: 0.8 Reporter: Swapnil Ghike Assignee: Swapnil Ghike Labels: improvement Fix For: 0.8 Attachments: kafka-987.patch, kafka-987-v2.patch We need to fix the Kafka zookeeper consumer to avoid checkpointing offsets that have not changed since the last offset commit. This will help reduce the write load on zookeeper. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Meetup in Raleigh/Durham, NC
Great, thanks for presenting and sharing this David! On Wed, Jul 24, 2013 at 6:16 AM, David Arthur mum...@gmail.com wrote: Here are the slides from the meetup http://www.slideshare.net/** mumrah/kafka-talk-tri-hughttp://www.slideshare.net/mumrah/kafka-talk-tri-hug We had 40-50 people show up which is about average for our meetup. Roughly half of the audience had heard of Kafka before this talk, and about 10 or so folks had used it or were using it in production. On 7/22/13 1:07 AM, Jun Rao wrote: David, Thanks for sharing this. Jun On Thu, Jul 18, 2013 at 9:19 PM, David Arthur mum...@gmail.com wrote: There is a Hadoop meetup happening in Durham next week. I'm presenting an intro to Kafka. I don't suspect there are any Kafka users in the area who are not already members of TriHUG, but sending this email out just in case :) http://www.meetup.com/TriHUG/events/129831822/http://www.meetup.com/TriHUG/**events/129831822/ http://www.**meetup.com/TriHUG/events/**129831822/http://www.meetup.com/TriHUG/events/129831822/ -David
[jira] [Updated] (KAFKA-957) MirrorMaker needs to preserve the key in the source cluster
[ https://issues.apache.org/jira/browse/KAFKA-957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-957: Attachment: KAFKA-957.v4.patch 7. Use Utils.abs to make sure the result of the hashCode function would be always non-negative. MirrorMaker needs to preserve the key in the source cluster --- Key: KAFKA-957 URL: https://issues.apache.org/jira/browse/KAFKA-957 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8 Reporter: Jun Rao Assignee: Guozhang Wang Attachments: KAFKA-957.v1.patch, KAFKA-957.v2.patch, KAFKA-957.v2.patch, KAFKA-957.v3.patch, KAFKA-957.v4.patch Currently, MirrorMaker only propagates the message to the target cluster, but not the associated key. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (KAFKA-982) Logo for Kafka
[ https://issues.apache.org/jira/browse/KAFKA-982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13715502#comment-13715502 ] Xavier Stevens edited comment on KAFKA-982 at 7/24/13 6:12 PM: --- +1 for 296, 298, 301 (updated to add 301) was (Author: xavier.stevens): +1 for 296, 298 Logo for Kafka -- Key: KAFKA-982 URL: https://issues.apache.org/jira/browse/KAFKA-982 Project: Kafka Issue Type: Improvement Reporter: Jay Kreps Attachments: 289.jpeg, 294.jpeg, 296.png, 298.jpeg, 301.png We should have a logo for kafka. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-982) Logo for Kafka
[ https://issues.apache.org/jira/browse/KAFKA-982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13718657#comment-13718657 ] Xavier Stevens commented on KAFKA-982: -- Maybe whoever did 301 could do another version where the lines represent morse code for Kafka? Logo for Kafka -- Key: KAFKA-982 URL: https://issues.apache.org/jira/browse/KAFKA-982 Project: Kafka Issue Type: Improvement Reporter: Jay Kreps Attachments: 289.jpeg, 294.jpeg, 296.png, 298.jpeg, 301.png We should have a logo for kafka. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-957) MirrorMaker needs to preserve the key in the source cluster
[ https://issues.apache.org/jira/browse/KAFKA-957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy updated KAFKA-957: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to 0.8 MirrorMaker needs to preserve the key in the source cluster --- Key: KAFKA-957 URL: https://issues.apache.org/jira/browse/KAFKA-957 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8 Reporter: Jun Rao Assignee: Guozhang Wang Attachments: KAFKA-957.v1.patch, KAFKA-957.v2.patch, KAFKA-957.v2.patch, KAFKA-957.v3.patch, KAFKA-957.v4.patch Currently, MirrorMaker only propagates the message to the target cluster, but not the associated key. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (KAFKA-957) MirrorMaker needs to preserve the key in the source cluster
[ https://issues.apache.org/jira/browse/KAFKA-957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy closed KAFKA-957. MirrorMaker needs to preserve the key in the source cluster --- Key: KAFKA-957 URL: https://issues.apache.org/jira/browse/KAFKA-957 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8 Reporter: Jun Rao Assignee: Guozhang Wang Attachments: KAFKA-957.v1.patch, KAFKA-957.v2.patch, KAFKA-957.v2.patch, KAFKA-957.v3.patch, KAFKA-957.v4.patch Currently, MirrorMaker only propagates the message to the target cluster, but not the associated key. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-957) MirrorMaker needs to preserve the key in the source cluster
[ https://issues.apache.org/jira/browse/KAFKA-957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13718690#comment-13718690 ] Joel Koshy commented on KAFKA-957: -- +1 MirrorMaker needs to preserve the key in the source cluster --- Key: KAFKA-957 URL: https://issues.apache.org/jira/browse/KAFKA-957 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8 Reporter: Jun Rao Assignee: Guozhang Wang Attachments: KAFKA-957.v1.patch, KAFKA-957.v2.patch, KAFKA-957.v2.patch, KAFKA-957.v3.patch, KAFKA-957.v4.patch Currently, MirrorMaker only propagates the message to the target cluster, but not the associated key. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-982) Logo for Kafka
[ https://issues.apache.org/jira/browse/KAFKA-982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Olson updated KAFKA-982: --- Attachment: 302.png Logo for Kafka -- Key: KAFKA-982 URL: https://issues.apache.org/jira/browse/KAFKA-982 Project: Kafka Issue Type: Improvement Reporter: Jay Kreps Attachments: 289.jpeg, 294.jpeg, 296.png, 298.jpeg, 301.png, 302.png We should have a logo for kafka. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-959) DefaultEventHandler can send more produce requests than necesary
[ https://issues.apache.org/jira/browse/KAFKA-959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-959: Status: Patch Available (was: Open) DefaultEventHandler can send more produce requests than necesary Key: KAFKA-959 URL: https://issues.apache.org/jira/browse/KAFKA-959 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8 Reporter: Jun Rao Assignee: Guozhang Wang Attachments: KAFKA-959.v1.patch In DefaultEventHandler, for a batch of messages, it picks a random partition per message (when there is no key specified). This means that it can send up to P produce requests where P is the number of partitions in a topic. A better way is probably to pick a single random partition for the whole batch of messages. This will reduce the number of produce requests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-959) DefaultEventHandler can send more produce requests than necesary
[ https://issues.apache.org/jira/browse/KAFKA-959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-959: Attachment: KAFKA-959.v1.patch Passed unit tests DefaultEventHandler can send more produce requests than necesary Key: KAFKA-959 URL: https://issues.apache.org/jira/browse/KAFKA-959 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8 Reporter: Jun Rao Assignee: Guozhang Wang Attachments: KAFKA-959.v1.patch In DefaultEventHandler, for a batch of messages, it picks a random partition per message (when there is no key specified). This means that it can send up to P produce requests where P is the number of partitions in a topic. A better way is probably to pick a single random partition for the whole batch of messages. This will reduce the number of produce requests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-984) Avoid a full rebalance in cases when a new topic is discovered but container/broker set stay the same
[ https://issues.apache.org/jira/browse/KAFKA-984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-984: Attachment: KAFKA-984.v2.patch Following Swapnil's comments, collapsing rebalance and rebalancePartial to reduce code duplication. Avoid a full rebalance in cases when a new topic is discovered but container/broker set stay the same - Key: KAFKA-984 URL: https://issues.apache.org/jira/browse/KAFKA-984 Project: Kafka Issue Type: Bug Reporter: Guozhang Wang Assignee: Guozhang Wang Fix For: 0.8 Attachments: KAFKA-984.v1.patch, KAFKA-984.v2.patch, KAFKA-984.v2.patch Currently a full rebalance will be triggered on high level consumers even when just a new topic is added to ZK. Better avoid this behavior but only rebalance on this newly added topic. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (KAFKA-988) Make ReassignReplica tool more usable
Sriram Subramanian created KAFKA-988: Summary: Make ReassignReplica tool more usable Key: KAFKA-988 URL: https://issues.apache.org/jira/browse/KAFKA-988 Project: Kafka Issue Type: Bug Reporter: Sriram Subramanian Assignee: Sriram Subramanian As part of this first iteration, we will have two options - The manual option takes a list of topic - partition - replicas list and reassigns them. The automatic option takes a list of topics and broker list to move the topics to. The tool assigns the replicas for the topic partitions to these brokers using the default assignment strategy. A dry run will be provided to see the assignment before actually doing the assignment. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
