[jira] [Commented] (KAFKA-982) Logo for Kafka
[ https://issues.apache.org/jira/browse/KAFKA-982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13716285#comment-13716285 ] Joel Koshy commented on KAFKA-982: -- +1 for 298, and I like Jakob's recursive suggestion as well (can you add feedback on that on the 99designs contest?). 294 seems interesting/deep (pen, two k's, I kind of see a person with hands raised, etc.), but I just prefer 298 wrt overall appearance. 296 is also good, but between 296 and 298 I prefer 298. Logo for Kafka -- Key: KAFKA-982 URL: https://issues.apache.org/jira/browse/KAFKA-982 Project: Kafka Issue Type: Improvement Reporter: Jay Kreps Attachments: 289.jpeg, 294.jpeg, 296.png, 298.jpeg We should have a logo for kafka. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Does Kafka message manager tool (read messages from console, delete messages) exist?
Hi, we use Kafka (0.7.2 version) in our product as message delivery service and message storage (our retention time 48h) system. Thanks a lot for perfect tool/lib. Now we really need 2 additional features: 1. Sometimes we need look at our messages we have in Kafka from command line, for example: ~$ kafkaTool get topic=my_topic_name from=timestamp_1 to=timestamp_2 | grep 'Exception' We need this feature because we want to have ability to see our raw messages body. 2. Sometimes we have to delete messages older than given timestamp. Or in other words we have to change offset for all consumers for given topic. (But we don't like to work with offset because we know only timestamp we like to move our consumers to). Example: ~$ kafkaTool purge topic=my_topic_name from=timestamp_1 to=timestamp_2 We need this feature because we want to skip processing of some messages by our regular consumers. Is these 2 features exist? If not, what is the best way to do this using the existing APIs? According to https://issues.apache.org/jira/browse/KAFKA-260 there is plan to add timestamp to message. Is it possible, or is it planned, to have ... something like FetchRequest, but with timestamp as parameter instead of offset? If this is currently not possible, should I open a JIRA issue? Best regards, Vladimir.
[jira] [Updated] (KAFKA-982) Logo for Kafka
[ https://issues.apache.org/jira/browse/KAFKA-982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-982: Attachment: 301.png Logo for Kafka -- Key: KAFKA-982 URL: https://issues.apache.org/jira/browse/KAFKA-982 Project: Kafka Issue Type: Improvement Reporter: Jay Kreps Attachments: 289.jpeg, 294.jpeg, 296.png, 298.jpeg, 301.png We should have a logo for kafka. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-982) Logo for Kafka
[ https://issues.apache.org/jira/browse/KAFKA-982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13716544#comment-13716544 ] Neha Narkhede commented on KAFKA-982: - +1 for 298, it's clean and simple. Second choice 296, I like the logging/pen reference. Logo for Kafka -- Key: KAFKA-982 URL: https://issues.apache.org/jira/browse/KAFKA-982 Project: Kafka Issue Type: Improvement Reporter: Jay Kreps Attachments: 289.jpeg, 294.jpeg, 296.png, 298.jpeg, 301.png We should have a logo for kafka. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-982) Logo for Kafka
[ https://issues.apache.org/jira/browse/KAFKA-982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13716562#comment-13716562 ] Neha Narkhede commented on KAFKA-982: - Oops, my bad. I meant 294 as my 2nd choice. Logo for Kafka -- Key: KAFKA-982 URL: https://issues.apache.org/jira/browse/KAFKA-982 Project: Kafka Issue Type: Improvement Reporter: Jay Kreps Attachments: 289.jpeg, 294.jpeg, 296.png, 298.jpeg, 301.png We should have a logo for kafka. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-879) In system test, read the new leader from zookeeper instead of broker log on completion of become-leader state transition
[ https://issues.apache.org/jira/browse/KAFKA-879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-879: -- Resolution: Fixed Fix Version/s: 0.8 Status: Resolved (was: Patch Available) Thanks for the patch. Committed to 0.8. In system test, read the new leader from zookeeper instead of broker log on completion of become-leader state transition Key: KAFKA-879 URL: https://issues.apache.org/jira/browse/KAFKA-879 Project: Kafka Issue Type: Bug Reporter: Swapnil Ghike Assignee: John Fung Labels: bugs Fix For: 0.8 Attachments: kafka-879-v1.patch Currently the system test reads the new leader from a broker log statement on completion of become-leader state transition. The log statements can change with time and can also move around. We can read the leader info from zookeeper given a topic, partition. The path is #2 at https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper. The Zookeeper data structures are unlikely to change in foreseeable future. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-982) Logo for Kafka
[ https://issues.apache.org/jira/browse/KAFKA-982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13716567#comment-13716567 ] Chris Riccomini commented on KAFKA-982: --- +1 301 Logo for Kafka -- Key: KAFKA-982 URL: https://issues.apache.org/jira/browse/KAFKA-982 Project: Kafka Issue Type: Improvement Reporter: Jay Kreps Attachments: 289.jpeg, 294.jpeg, 296.png, 298.jpeg, 301.png We should have a logo for kafka. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-922) System Test - set retry.backoff.ms=300 to testcase_0119
[ https://issues.apache.org/jira/browse/KAFKA-922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-922: -- Resolution: Fixed Fix Version/s: 0.8 Assignee: John Fung Status: Resolved (was: Patch Available) This is already committed to 0.8. System Test - set retry.backoff.ms=300 to testcase_0119 --- Key: KAFKA-922 URL: https://issues.apache.org/jira/browse/KAFKA-922 Project: Kafka Issue Type: Task Reporter: John Fung Assignee: John Fung Fix For: 0.8 Attachments: kafka-922-v1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (KAFKA-922) System Test - set retry.backoff.ms=300 to testcase_0119
[ https://issues.apache.org/jira/browse/KAFKA-922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao closed KAFKA-922. - System Test - set retry.backoff.ms=300 to testcase_0119 --- Key: KAFKA-922 URL: https://issues.apache.org/jira/browse/KAFKA-922 Project: Kafka Issue Type: Task Reporter: John Fung Assignee: John Fung Fix For: 0.8 Attachments: kafka-922-v1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-932) System Test - set retry.backoff.ms=300 to all test cases
[ https://issues.apache.org/jira/browse/KAFKA-932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-932: -- Resolution: Fixed Fix Version/s: 0.8 Assignee: John Fung Status: Resolved (was: Patch Available) Thanks for the patch. Committed to 0.8. System Test - set retry.backoff.ms=300 to all test cases Key: KAFKA-932 URL: https://issues.apache.org/jira/browse/KAFKA-932 Project: Kafka Issue Type: Task Reporter: John Fung Assignee: John Fung Fix For: 0.8 Attachments: kafka-932-v1.patch 1. Set retry.backoff.ms=300 to all test cases 2. Set each bouncing 1 minute apart in replication_testsuite -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-987) Avoid checkpointing offsets in Kafka consumer that have not changed since the last commit
[ https://issues.apache.org/jira/browse/KAFKA-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13716583#comment-13716583 ] Neha Narkhede commented on KAFKA-987: - Thanks for the patch, Swapnil. Good thinking about not limiting the check to offset committed offset. Just one question about your patch - In commitOffsets, should the the map update move to inside the try block to ensure that the map is updated only if the zk write succeeds ? Avoid checkpointing offsets in Kafka consumer that have not changed since the last commit - Key: KAFKA-987 URL: https://issues.apache.org/jira/browse/KAFKA-987 Project: Kafka Issue Type: Bug Affects Versions: 0.8 Reporter: Swapnil Ghike Assignee: Swapnil Ghike Labels: improvement Fix For: 0.8 Attachments: kafka-987.patch We need to fix the Kafka zookeeper consumer to avoid checkpointing offsets that have not changed since the last offset commit. This will help reduce the write load on zookeeper. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-925) Add optional partition key override in producer
[ https://issues.apache.org/jira/browse/KAFKA-925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13716593#comment-13716593 ] Joel Koshy commented on KAFKA-925: -- +1 , looks good to me. DefaultPartitioner: Do we need the type parameter anymore? Guozhang has a good point about tools such as mirror maker not having access to the original partitioning key. However, I can see that it would be clunky as we would then need a partition key serializer as well. Also, for something like offset-preserving mirrors we would anyway have the source cluster's partition available, so I don't see it as a major issue. ConsoleProducer: the enqueue timeout change seems reasonable - I'm assuming it was done to avoid dropping messages when piping into ConsoleProducer. Correct? Add optional partition key override in producer --- Key: KAFKA-925 URL: https://issues.apache.org/jira/browse/KAFKA-925 Project: Kafka Issue Type: New Feature Components: producer Affects Versions: 0.8.1 Reporter: Jay Kreps Assignee: Jay Kreps Attachments: KAFKA-925-v1.patch, KAFKA-925-v2.patch We have a key that is used for partitioning in the producer and stored with the message. Actually these uses, though often the same, could be different. The two meanings are effectively: 1. Assignment to a partition 2. Deduplication within a partition In cases where we want to allow the client to take advantage of both of these and they aren't the same it would be nice to allow them to be specified separately. To implement this I added an optional partition key to KeyedMessage. When specified this key is used for partitioning rather than the message key. This key is of type Any and the parametric typing is removed from the partitioner to allow it to work with either key. An alternative would be to allow the partition id to specified in the KeyedMessage. This would be slightly more convenient in the case where there is no partition key but instead you know a priori the partition number--this case must be handled by giving the partition id as the partition key and using an identity partitioner which is slightly more roundabout. However this is inconsistent with the normal partitioning which requires a key in the case where the partition is determined by a key--in that case you would be manually calling your partitioner in user code. It seems best to me to either use a key or always a partition and since we currently take a key I stuck with that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-982) Logo for Kafka
[ https://issues.apache.org/jira/browse/KAFKA-982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13716601#comment-13716601 ] Jay Kreps commented on KAFKA-982: - Not sure that I get 301, what are the pictures? They kind of look like sideways i's... Logo for Kafka -- Key: KAFKA-982 URL: https://issues.apache.org/jira/browse/KAFKA-982 Project: Kafka Issue Type: Improvement Reporter: Jay Kreps Attachments: 289.jpeg, 294.jpeg, 296.png, 298.jpeg, 301.png We should have a logo for kafka. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-982) Logo for Kafka
[ https://issues.apache.org/jira/browse/KAFKA-982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13716607#comment-13716607 ] David Arthur commented on KAFKA-982: [~jkreps], to me they look like messages in a log file. Like the straight lines are the messages and the dots are delimiters. Then overall, it's shaped like a K (maybe?) Logo for Kafka -- Key: KAFKA-982 URL: https://issues.apache.org/jira/browse/KAFKA-982 Project: Kafka Issue Type: Improvement Reporter: Jay Kreps Attachments: 289.jpeg, 294.jpeg, 296.png, 298.jpeg, 301.png We should have a logo for kafka. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-882) Enhance 0.7 ProducerPerformance to send sequential MessageID as in 0.8
[ https://issues.apache.org/jira/browse/KAFKA-882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-882: -- Resolution: Fixed Fix Version/s: 0.7 Status: Resolved (was: Patch Available) Thanks for the patch. Committed to 0.7 so that we can get the system tests in 0.8 working. Enhance 0.7 ProducerPerformance to send sequential MessageID as in 0.8 -- Key: KAFKA-882 URL: https://issues.apache.org/jira/browse/KAFKA-882 Project: Kafka Issue Type: Task Reporter: John Fung Assignee: John Fung Labels: kafka Fix For: 0.7 Attachments: kafka-882-v1.patch Also update related files in the following in Kafka 0.8: 1. system_test/migration_tool_testsuite/config 2. system_test/migration_tool_testsuite/0.7/lib/kafka-perf-0.7.0.jar (after this JIRA is completed) 3. system_test/migration_tool_testsuite/testcase_900x/testcase_900x_properties.json -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-984) Avoid a full rebalance in cases when a new topic is discovered but container/broker set stay the same
[ https://issues.apache.org/jira/browse/KAFKA-984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-984: Status: Patch Available (was: Open) Avoid a full rebalance in cases when a new topic is discovered but container/broker set stay the same - Key: KAFKA-984 URL: https://issues.apache.org/jira/browse/KAFKA-984 Project: Kafka Issue Type: Bug Reporter: Guozhang Wang Assignee: Guozhang Wang Fix For: 0.8 Attachments: KAFKA-984.v1.patch Currently a full rebalance will be triggered on high level consumers even when just a new topic is added to ZK. Better avoid this behavior but only rebalance on this newly added topic. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-984) Avoid a full rebalance in cases when a new topic is discovered but container/broker set stay the same
[ https://issues.apache.org/jira/browse/KAFKA-984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-984: Attachment: KAFKA-984.v1.patch Avoid a full rebalance in cases when a new topic is discovered but container/broker set stay the same - Key: KAFKA-984 URL: https://issues.apache.org/jira/browse/KAFKA-984 Project: Kafka Issue Type: Bug Reporter: Guozhang Wang Assignee: Guozhang Wang Fix For: 0.8 Attachments: KAFKA-984.v1.patch Currently a full rebalance will be triggered on high level consumers even when just a new topic is added to ZK. Better avoid this behavior but only rebalance on this newly added topic. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-987) Avoid checkpointing offsets in Kafka consumer that have not changed since the last commit
[ https://issues.apache.org/jira/browse/KAFKA-987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Swapnil Ghike updated KAFKA-987: Attachment: kafka-987-v2.patch Yes, thank you. Attached v2. Avoid checkpointing offsets in Kafka consumer that have not changed since the last commit - Key: KAFKA-987 URL: https://issues.apache.org/jira/browse/KAFKA-987 Project: Kafka Issue Type: Bug Affects Versions: 0.8 Reporter: Swapnil Ghike Assignee: Swapnil Ghike Labels: improvement Fix For: 0.8 Attachments: kafka-987.patch, kafka-987-v2.patch We need to fix the Kafka zookeeper consumer to avoid checkpointing offsets that have not changed since the last offset commit. This will help reduce the write load on zookeeper. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-987) Avoid checkpointing offsets in Kafka consumer that have not changed since the last commit
[ https://issues.apache.org/jira/browse/KAFKA-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13716680#comment-13716680 ] Neha Narkhede commented on KAFKA-987: - +1 on v2. Avoid checkpointing offsets in Kafka consumer that have not changed since the last commit - Key: KAFKA-987 URL: https://issues.apache.org/jira/browse/KAFKA-987 Project: Kafka Issue Type: Bug Affects Versions: 0.8 Reporter: Swapnil Ghike Assignee: Swapnil Ghike Labels: improvement Fix For: 0.8 Attachments: kafka-987.patch, kafka-987-v2.patch We need to fix the Kafka zookeeper consumer to avoid checkpointing offsets that have not changed since the last offset commit. This will help reduce the write load on zookeeper. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (KAFKA-987) Avoid checkpointing offsets in Kafka consumer that have not changed since the last commit
[ https://issues.apache.org/jira/browse/KAFKA-987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede resolved KAFKA-987. - Resolution: Fixed Avoid checkpointing offsets in Kafka consumer that have not changed since the last commit - Key: KAFKA-987 URL: https://issues.apache.org/jira/browse/KAFKA-987 Project: Kafka Issue Type: Bug Affects Versions: 0.8 Reporter: Swapnil Ghike Assignee: Swapnil Ghike Labels: improvement Fix For: 0.8 Attachments: kafka-987.patch, kafka-987-v2.patch We need to fix the Kafka zookeeper consumer to avoid checkpointing offsets that have not changed since the last offset commit. This will help reduce the write load on zookeeper. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-987) Avoid checkpointing offsets in Kafka consumer that have not changed since the last commit
[ https://issues.apache.org/jira/browse/KAFKA-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13716683#comment-13716683 ] Neha Narkhede commented on KAFKA-987: - committed v2 to 0.8 Avoid checkpointing offsets in Kafka consumer that have not changed since the last commit - Key: KAFKA-987 URL: https://issues.apache.org/jira/browse/KAFKA-987 Project: Kafka Issue Type: Bug Affects Versions: 0.8 Reporter: Swapnil Ghike Assignee: Swapnil Ghike Labels: improvement Fix For: 0.8 Attachments: kafka-987.patch, kafka-987-v2.patch We need to fix the Kafka zookeeper consumer to avoid checkpointing offsets that have not changed since the last offset commit. This will help reduce the write load on zookeeper. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-982) Logo for Kafka
[ https://issues.apache.org/jira/browse/KAFKA-982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13716697#comment-13716697 ] Mahendra M commented on KAFKA-982: -- +296, 298 Logo for Kafka -- Key: KAFKA-982 URL: https://issues.apache.org/jira/browse/KAFKA-982 Project: Kafka Issue Type: Improvement Reporter: Jay Kreps Attachments: 289.jpeg, 294.jpeg, 296.png, 298.jpeg, 301.png We should have a logo for kafka. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-982) Logo for Kafka
[ https://issues.apache.org/jira/browse/KAFKA-982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13717618#comment-13717618 ] Benjamin Black commented on KAFKA-982: -- None of them make much sense, but 296 is least bad. Logo for Kafka -- Key: KAFKA-982 URL: https://issues.apache.org/jira/browse/KAFKA-982 Project: Kafka Issue Type: Improvement Reporter: Jay Kreps Attachments: 289.jpeg, 294.jpeg, 296.png, 298.jpeg, 301.png We should have a logo for kafka. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-984) Avoid a full rebalance in cases when a new topic is discovered but container/broker set stay the same
[ https://issues.apache.org/jira/browse/KAFKA-984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-984: Attachment: KAFKA-984.v2.patch Added one more unit test in ZookeeperConsumerConnectorTest for wildcard consumer. Avoid a full rebalance in cases when a new topic is discovered but container/broker set stay the same - Key: KAFKA-984 URL: https://issues.apache.org/jira/browse/KAFKA-984 Project: Kafka Issue Type: Bug Reporter: Guozhang Wang Assignee: Guozhang Wang Fix For: 0.8 Attachments: KAFKA-984.v1.patch, KAFKA-984.v2.patch Currently a full rebalance will be triggered on high level consumers even when just a new topic is added to ZK. Better avoid this behavior but only rebalance on this newly added topic. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-982) Logo for Kafka
[ https://issues.apache.org/jira/browse/KAFKA-982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13717807#comment-13717807 ] Jay Kreps commented on KAFKA-982: - Okay, someone just pointed out that 294 is...uh...kind of vulvic. I think this person is clearly just a pervert, but I think I want to change my vote all the same :-) Logo for Kafka -- Key: KAFKA-982 URL: https://issues.apache.org/jira/browse/KAFKA-982 Project: Kafka Issue Type: Improvement Reporter: Jay Kreps Attachments: 289.jpeg, 294.jpeg, 296.png, 298.jpeg, 301.png We should have a logo for kafka. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-984) Avoid a full rebalance in cases when a new topic is discovered but container/broker set stay the same
[ https://issues.apache.org/jira/browse/KAFKA-984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13717894#comment-13717894 ] Swapnil Ghike commented on KAFKA-984: - Hey Guozhang, this looks like a reasonable approach. But we should probably do some code de-dup on this, let's talk tomorrow in person. Avoid a full rebalance in cases when a new topic is discovered but container/broker set stay the same - Key: KAFKA-984 URL: https://issues.apache.org/jira/browse/KAFKA-984 Project: Kafka Issue Type: Bug Reporter: Guozhang Wang Assignee: Guozhang Wang Fix For: 0.8 Attachments: KAFKA-984.v1.patch, KAFKA-984.v2.patch Currently a full rebalance will be triggered on high level consumers even when just a new topic is added to ZK. Better avoid this behavior but only rebalance on this newly added topic. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Does Kafka message manager tool (read messages from console, delete messages) exist?
For 1, you can write a tool that uses our getOffsetBefore api and SimpleConsumer. The offset returned is at the log segment boundary. So it's not going to match the specified time precisely. For 2, you can configure a topic to be rolled by timestamp (log.roll.hours and topic.log.roll.hours). Then set the retention time accordingly. Thanks, Jun On Tue, Jul 23, 2013 at 5:21 AM, Vladimir Tretyakov vladimir.tretya...@sematext.com wrote: Hi, we use Kafka (0.7.2 version) in our product as message delivery service and message storage (our retention time 48h) system. Thanks a lot for perfect tool/lib. Now we really need 2 additional features: 1. Sometimes we need look at our messages we have in Kafka from command line, for example: ~$ kafkaTool get topic=my_topic_name from=timestamp_1 to=timestamp_2 | grep 'Exception' We need this feature because we want to have ability to see our raw messages body. 2. Sometimes we have to delete messages older than given timestamp. Or in other words we have to change offset for all consumers for given topic. (But we don't like to work with offset because we know only timestamp we like to move our consumers to). Example: ~$ kafkaTool purge topic=my_topic_name from=timestamp_1 to=timestamp_2 We need this feature because we want to skip processing of some messages by our regular consumers. Is these 2 features exist? If not, what is the best way to do this using the existing APIs? According to https://issues.apache.org/jira/browse/KAFKA-260 there is plan to add timestamp to message. Is it possible, or is it planned, to have ... something like FetchRequest, but with timestamp as parameter instead of offset? If this is currently not possible, should I open a JIRA issue? Best regards, Vladimir.
[jira] [Commented] (KAFKA-987) Avoid checkpointing offsets in Kafka consumer that have not changed since the last commit
[ https://issues.apache.org/jira/browse/KAFKA-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13717977#comment-13717977 ] Jun Rao commented on KAFKA-987: --- It seems that we don't need to update the offset map in addPartitionTopicInfo(). In fact, currently, if there is no new messages coming in, we won't checkpoint the first offset. Avoid checkpointing offsets in Kafka consumer that have not changed since the last commit - Key: KAFKA-987 URL: https://issues.apache.org/jira/browse/KAFKA-987 Project: Kafka Issue Type: Bug Affects Versions: 0.8 Reporter: Swapnil Ghike Assignee: Swapnil Ghike Labels: improvement Fix For: 0.8 Attachments: kafka-987.patch, kafka-987-v2.patch We need to fix the Kafka zookeeper consumer to avoid checkpointing offsets that have not changed since the last offset commit. This will help reduce the write load on zookeeper. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
